Skip to content

Path 7 · lesson 3 of 8

Advanced35 minApache Kafka 4.x (configuration reference read for 4.1) · Spring for Apache Kafka 3.3 on Spring Boot 3.x · Java 21 · PostgreSQL per service

The saga pattern

A business transaction across services as a chain of local transactions, each with a compensating action. Choreography vs orchestration, idempotency, timeouts and living without isolation.

What you will learn

  • Break a cross-service business transaction into local transactions and define the compensating action for each
  • Compare choreography and orchestration and choose between them for a given workflow
  • Implement a choreographed order saga in Spring Boot in which a failed payment cancels the order and rejected stock refunds the payment
  • Make every saga step idempotent and state-guarded so duplicates, late events and repeated compensations are harmless
  • Add a timeout so a saga cannot stay unfinished, and handle events that arrive after it fired
  • Explain what the missing isolation means for readers of intermediate state, and apply countermeasures

Before you start

The concept

Placing an order changes three databases: the Order Service records the order, the Payment Service takes the money, the Inventory Service reserves the stock. Each service owns its database, so no single transaction can cover all three. A saga achieves the business outcome without one: it is a sequence of local transactions, one per service, linked by messages. Each step commits on its own and then triggers the next.

Because earlier steps are already committed when a later one fails, there is nothing to roll back. Instead, every step that can need undoing has a compensating action: another local transaction that semantically reverses it. If stock cannot be reserved after the payment was authorised, the saga runs the payment's compensation, a refund, and then cancels the order. The saga ends in one of two consistent outcomes: every step done, or every completed step compensated.

StepServiceLocal transactionCompensating action
1OrderCreate order as PENDINGMark order CANCELLED
2PaymentAuthorise paymentRefund (or void) the payment
3InventoryReserve stockRelease the reservation
4OrderMark order CONFIRMEDNone: nothing follows that can fail
5NotificationSend confirmation or cancellationNone: an email cannot be unsent, so it goes last
The order saga used in this lesson.

Two ways to coordinate

Choreography: services react to events

  • Each service subscribes to the others' events and publishes its own. There is no central component.
  • Adding a reader such as Notification needs no change to existing services
  • No single point of failure and no extra service to run
  • The workflow is not written down anywhere: it emerges from who listens to what
  • Hard to see where one order is stuck without tracing events across services
  • Cyclic dependencies appear as the flow grows: services must know each other's events

Orchestration: a coordinator sends commands

  • An orchestrator holds the saga's state and tells each participant what to do, then waits for the reply
  • The whole flow, including compensations and timeouts, is in one place and one state table
  • Participants stay simple: they execute commands and report results
  • Easy to answer 'what state is order 1001 in?' and to add steps in the middle
  • The orchestrator is one more service to build, run and keep available
  • Risk of pulling business logic out of the services into the coordinator

A rule of thumb that holds up: choreography suits a short, stable flow of three or four steps with few branches. Once a flow has many steps, several failure branches, or timeouts between steps, the implicit version becomes difficult to reason about and an orchestrator pays for itself. Both use the same building blocks from this path: an outbox to publish, idempotent consumers to receive.

Advantages of a saga

  • Keeps data consistent across services without a distributed transaction
  • Each service stays available independently: a slow participant delays the saga without blocking the others' databases
  • No locks held across services or across network calls
  • Every step is an ordinary local transaction that the owning team can test alone

Disadvantages of a saga

  • No isolation: intermediate states are visible to other requests and other sagas
  • No automatic rollback: every compensation is code you design, write and test
  • Eventually consistent: the caller gets 'pending' and learns the outcome later
  • Needs idempotency, timeouts and monitoring for stuck sagas to be correct
  • Debugging means following one business transaction through several logs

Why it exists

In a single database the order flow is one transaction: insert the order, debit the account, decrement the stock, commit. If the stock check fails, everything rolls back and no other session ever saw the half-finished state. Splitting the system into services with their own databases removes that option, and the obvious replacements both fail.

  • Two-phase commit across the services' databases. It needs every participant to support it and to be available at the same moment. While a participant has voted yes and is waiting for the decision, it holds locks. If the coordinator disappears at that point, the locks stay. Kafka cannot take part in such a transaction, and neither can an external payment provider.
  • A chain of synchronous calls with try/catch. The Order Service calls Payment, then Inventory, and on an exception calls Payment again to refund. If the Order Service crashes after the payment call returned, nothing remembers that a refund is owed. If the refund call times out, was it applied? The undo logic lives in the memory of a process that can die.

The saga idea comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem on long-lived database transactions: split the long transaction into a sequence of shorter ones that each commit, and pair each with a compensating transaction, so the system either completes the whole sequence or runs the compensations of the completed part. Applied to services, the sequence is driven by durable messages instead of one process. That is the difference from the try/catch chain: the fact that a step happened is stored in a database and announced on a durable log, so the next step, or the compensation, happens even if every service restarts in between.

Kafka fits this role well. Events are retained, so a consumer that was down picks up where it stopped. Keying every saga event by order id keeps the events of one saga on one topic in order. Consumer groups let each service process its share independently. What Kafka does not provide is atomicity between a service's database and its events. Each saga step must update local state and publish its result atomically, which is exactly what the transactional outbox is for.

How it works inside

The shape of one step

Every participant does the same four things in one local database transaction: record the incoming event id (skip if already seen), check that its own state allows the transition, change its state, and write the outgoing event to its outbox. The offset is committed after the transaction. A crash anywhere leaves either nothing done (the event is redelivered and processed) or everything done (the redelivery is skipped).

State guards make steps and compensations idempotent

Deduplication by event id handles the same event delivered twice. It does not handle two different events that ask for the same thing: a refund can be triggered by StockRejected and also by OrderCancelled from the timeout. The second defence is a guarded transition: UPDATE payment SET status = 'REFUNDED' WHERE order_id = ? AND status = 'AUTHORIZED'. Only one statement can ever match. Whoever gets zero rows does nothing and publishes nothing.

Drawing diagram…

State diagram of an order: it starts PENDING, becomes CONFIRMED on StockReserved, or CANCELLED on PaymentFailed, PaymentRefunded or a timeout. CONFIRMED and CANCELLED are final.

stateDiagram-v2
  [*] --> PENDING: order placed
  PENDING --> CONFIRMED: StockReserved
  PENDING --> CANCELLED: PaymentFailed
  PENDING --> CANCELLED: PaymentRefunded
  PENDING --> CANCELLED: timeout
  CONFIRMED --> [*]
  CANCELLED --> [*]
The order is the saga's visible state. Every transition leaves PENDING exactly once.

No isolation, and how to live with it

Between step 1 and step 4 the order exists, the money may be held, and the stock may or may not be reserved. Anyone can observe that. A second saga can read stock that a first saga is about to give back. An ACID transaction would hide all of this, and a saga cannot. The countermeasures are design choices:

  • Semantic lock. The PENDING status is a flag that says 'a saga is working on this'. The UI shows 'processing', other operations (edit, cancel by the customer, ship) check the status and refuse or wait.
  • Reserve, do not consume. Authorise the payment and capture later. Reserve stock and deduct on shipment. Reservations are cheap to compensate and honest about being provisional.
  • Guard on current state, not on what you read earlier. available = available - ? WHERE available >= ? decides at write time, so two sagas competing for the last item cannot both win.
  • Order the steps by how hard they are to undo. Put steps that can fail for business reasons early and steps that cannot be compensated (sending an email, shipping a parcel) last, after the point where the saga can no longer fail.
  • Make compensations commutative with late events. A compensation may run before the step it compensates has even been processed by a slow service. Record the cancellation so the late step is ignored.

Timeouts

A choreographed saga has no component waiting for the next event. If the Payment Service is down for a day, the order simply stays PENDING. The Order Service therefore runs a sweeper: orders still PENDING after a deadline are cancelled and OrderCancelled is published. Every participant treats that event as a request to compensate whatever it has done. The timeout does not stop the other services, so an event can arrive after it: the payment is authorised a minute after the order was cancelled. Because OrderCreated and OrderCancelled are on the same topic with the same key, the Payment Service sees them in order and its guarded refund handles it.

Failure scenarios

Failure pointResulting stateRecovery
Payment Service crashes after authorising at the provider, before its commitProvider holds the money. No payment row update, no event, offset not committedOrderCreated is redelivered. The authorise call is repeated with the same idempotency key (the order id), the provider returns the original result, the step commits.
Payment Service crashes after its commit, before the offset commitPayment AUTHORIZED, PaymentAuthorized in its outboxThe redelivered OrderCreated is skipped by event id. The relay publishes the outbox row.
Payment is declinedPayment FAILED, PaymentFailed publishedOrder Service cancels the order. Nothing else was done, so nothing else is compensated.
Stock is not availablePayment AUTHORIZED, stock request REJECTED, order still PENDINGPayment Service refunds on StockRejected and publishes PaymentRefunded. Order Service then cancels.
The refund call to the provider failsPayment still AUTHORIZED: the transaction rolled backThe error handler redelivers StockRejected with backoff. If retries are exhausted the record goes to a dead letter topic and a person is alerted: money is owed.
Inventory receives PaymentAuthorized before it has processed OrderCreatedNo stock request row to act onThe listener throws, the record is retried after a backoff, by which time the other partition has usually caught up.
A participant is down for longer than the saga deadlineOrder PENDING past the deadlineThe sweeper cancels and publishes OrderCancelled. When the participant returns it processes its backlog in order and compensates.
Order Service crashes between any two eventsOrder PENDING, events waiting on the topicsOn restart it resumes from its committed offsets. Saga state is in the database and the log, never in memory.
Crash or failure at each point, the state it leaves, and how the saga recovers.

Architecture

Drawing diagram…

Flowchart: Order, Payment and Inventory services each publish to their own topic, orders.events, payments.events and inventory.events. Payment and Inventory and Notification consume orders.events. Inventory and Order consume payments.events. Payment and Order consume inventory.events. The Payment Service also calls an external payment provider with an idempotency key.

flowchart LR
  OS["Order Service"]
  PS["Payment Service"]
  IS["Inventory Service"]
  NS["Notification Service"]
  OE["Topic orders.events"]
  PE["Topic payments.events"]
  IE["Topic inventory.events"]
  OS -->|"OrderCreated, OrderConfirmed, OrderCancelled"| OE
  PS -->|"PaymentAuthorized, PaymentFailed, PaymentRefunded"| PE
  IS -->|"StockReserved, StockRejected"| IE
  OE --> PS
  OE --> IS
  OE --> NS
  PE --> IS
  PE --> OS
  IE --> PS
  IE --> OS
  PSP["Payment provider"]
  PS -.->|"authorise, refund with idempotency key"| PSP
Choreography: each service publishes to its own topic through its outbox and subscribes to the topics it reacts to. Every record is keyed by order id.
  • Each service is the only writer of its topic, and writes through its outbox, so a state change and its event cannot diverge.
  • The arrows into each service are its subscriptions. The workflow is the sum of those subscriptions, which is why choreography is hard to read from any single codebase.
  • The order id is the key on all three topics. Within one topic the events of one order are ordered. Across topics they are not: Inventory can see PaymentAuthorized before OrderCreated.
  • The payment provider is outside every transaction. The only protection for that call is an idempotency key that makes repeating it safe.

Message flow

Drawing diagram…

Sequence diagram: the Order Service publishes OrderCreated. Inventory stores the request and Payment authorises and publishes PaymentAuthorized. Inventory finds no stock and publishes StockRejected. Payment refunds and publishes PaymentRefunded. The Order Service cancels the order and publishes OrderCancelled. Notification emails the customer, and Payment receives OrderCancelled and does nothing because the payment is already refunded.

sequenceDiagram
  participant O as Order Service
  participant K as Kafka
  participant P as Payment Service
  participant I as Inventory Service
  participant N as Notification Service
  O->>K: OrderCreated o-1001
  K->>P: OrderCreated
  K->>I: OrderCreated
  I->>I: store stock request RECEIVED
  P->>P: authorise, payment AUTHORIZED
  P->>K: PaymentAuthorized
  K->>I: PaymentAuthorized
  I->>I: no stock left, request REJECTED
  I->>K: StockRejected
  K->>P: StockRejected
  P->>P: refund, payment REFUNDED
  P->>K: PaymentRefunded
  K->>O: PaymentRefunded
  O->>O: order PENDING to CANCELLED
  O->>K: OrderCancelled
  K->>N: OrderCancelled
  N->>N: email order cancelled, money returned
  K->>P: OrderCancelled
  P->>P: already REFUNDED, nothing to do
The compensation path: stock is rejected after the payment was authorised, so the payment is refunded and then the order is cancelled.
  1. Compensations run in reverse order of the steps: stock was never reserved, so only the payment and the order are undone.
  2. The order is cancelled on PaymentRefunded, not on StockRejected. When the customer is told 'cancelled', the refund has already been issued.
  3. The last message shows why guards matter: OrderCancelled also asks Payment to compensate, and the guarded update finds the payment already refunded.
  4. In the declined-payment path the sequence is shorter: PaymentFailed goes straight to the Order Service, which cancels. Inventory never acts.

In a real system

A typical shop runs this saga on three topics with 12 partitions each, replication factor 3, all keyed by order id: orders.events, payments.events, inventory.events. Consumer groups are named after the services: order-service reads payments and inventory events, payment-service reads orders and inventory events, inventory-service reads orders and payments events, notification-service reads orders events. The checkout API returns 202 Accepted with the order id while the order is PENDING. The client polls GET /orders/{id} or receives a push when the status changes.

Question a user or operator asksWhere the answer lives
Did my order go through?The order's status in the Order Service: PENDING, CONFIRMED or CANCELLED with a reason
Why is order 1001 still pending?The last event for key o-1001 on each topic, and the lag of the group that should have reacted
Was the customer refunded?The payment row's status and the PaymentRefunded event
How many sagas are stuck?Count of orders PENDING for longer than the deadline: this should be zero and is the saga's main alert

When to use it

  • A business transaction changes data owned by several services, and the outcome must be all-or-compensated.
  • Every step that can fail has a meaningful compensation the business accepts (refund, release, cancel).
  • The caller can accept an asynchronous result: 'pending' now, the outcome shortly after.
  • Choreography when the flow is short and stable. Orchestration when it has many steps, branches or waiting periods.

When not to use it

  • All the data sits in one database. Use a local transaction: it is simpler and gives real isolation.
  • A step cannot be compensated and cannot be moved to the end, and the business cannot tolerate it being wrong.
  • The caller needs a consistent answer before the request returns and cannot work with a pending state.
  • The services are split so that almost every request needs a saga. That usually means the service boundaries are in the wrong place.

Java and Spring Boot

A choreographed order saga on Spring Boot 3 with one PostgreSQL database per service. Every listener runs in a local transaction that deduplicates the incoming event, applies a guarded state change and writes the outgoing event to the service's outbox (published by the relay or CDC connector from the previous two lessons). All records carry the order id as key and the headers id and eventType.

schema.sql (one section per service database)SQL
-- Every service database has these two tables (see the transactional outbox lesson).CREATE TABLE outbox_event (    seq           bigserial    PRIMARY KEY,    id            uuid         NOT NULL UNIQUE,    aggregatetype varchar(255) NOT NULL,      -- 'orders' | 'payments' | 'inventory' -> <type>.events    aggregateid   varchar(255) NOT NULL,      -- always the order id: the saga's correlation key    type          varchar(255) NOT NULL,    payload       jsonb        NOT NULL,    published_at  timestamptz);CREATE TABLE processed_event (    consumer  varchar(64) NOT NULL,    event_id  uuid        NOT NULL,    PRIMARY KEY (consumer, event_id));-- Order ServiceCREATE TABLE orders (    id            varchar(36) PRIMARY KEY,    customer_id   varchar(36) NOT NULL,    sku           varchar(64) NOT NULL,    quantity      integer     NOT NULL,    total_cents   bigint      NOT NULL,    status        varchar(16) NOT NULL,       -- PENDING | CONFIRMED | CANCELLED    cancel_reason varchar(64),    created_at    timestamptz NOT NULL DEFAULT now());-- Guarded transition, used for every status change (1 row = this caller won):--   UPDATE orders SET status = 'CANCELLED', cancel_reason = ? WHERE id = ? AND status = 'PENDING';-- Timeout sweep:--   SELECT id FROM orders WHERE status = 'PENDING' AND created_at < now() - interval '15 minutes';-- Payment ServiceCREATE TABLE payment (    order_id     varchar(36) PRIMARY KEY,     -- one payment per order    amount_cents bigint      NOT NULL,    status       varchar(16) NOT NULL         -- PENDING | AUTHORIZED | FAILED | REFUNDED);-- Compensation, safe to request any number of times:--   UPDATE payment SET status = 'REFUNDED' WHERE order_id = ? AND status = 'AUTHORIZED';-- Inventory ServiceCREATE TABLE stock_level (    sku       varchar(64) PRIMARY KEY,    available integer     NOT NULL CHECK (available >= 0));CREATE TABLE stock_request (    order_id varchar(36) PRIMARY KEY,    sku      varchar(64) NOT NULL,    quantity integer     NOT NULL,    status   varchar(16) NOT NULL             -- RECEIVED | RESERVED | REJECTED | RELEASED | CANCELLED);-- Reserve without overselling (decided at write time, not from an earlier read):--   UPDATE stock_level SET available = available - ? WHERE sku = ? AND available >= ?;-- Compensation:--   UPDATE stock_level SET available = available + ? WHERE sku = ?;

Why it is written this way

  • Each step's transaction touches only that service's tables: processed_event, one business table, outbox_event. No statement crosses a service boundary.
  • aggregateid is the order id in all three outboxes, including payments and inventory. It becomes the record key, so all events of one saga on one topic stay in order.
  • The status columns are the saga's state. There is no separate saga table in a choreographed design, which is part of why the overall flow is hard to see.
  • The available >= ? condition is the isolation countermeasure for stock: two concurrent sagas cannot both take the last unit, whatever each of them read earlier.

Configuration

A saga's correctness comes from the code above, but these settings decide how it behaves under failure. Kafka defaults are from the 4.1 consumer configuration reference, Spring behaviour from the Spring for Apache Kafka 3.3 reference. The last two are application settings.

enable.auto.commitDefault true in the Kafka consumer. Spring for Apache Kafka sets it to false unless you configure it
What it does
Whether the consumer commits offsets periodically in the background, regardless of whether processing finished.
When to change it
Leave Spring's false. The listener container then commits after the listener returns, which is what makes a saga step at-least-once.
spring.kafka.listener.ack-modeDefault BATCH
What it does
When the container commits: BATCH commits after all records returned by one poll have been processed.
When to change it
The default is fine with idempotent steps. RECORD commits after each record and reduces the number of redeliveries after a crash, at the cost of more commits.
max.poll.interval.msDefault 300000 (5 minutes)
What it does
Maximum time between two calls to poll() before the consumer is considered failed and its partitions are reassigned.
When to change it
Processing of one poll's records, including in-thread retries and provider calls, must fit within it. Lower max.poll.records (default 500) or shorten timeouts before raising this.
auto.offset.resetDefault latest
What it does
Where a group with no committed offset starts reading.
When to change it
Set earliest for saga participants. With latest, a new service or a renamed group silently skips events that were published before it first connected.
DefaultErrorHandler back-offDefault FixedBackOff(0L, 9): nine immediate retries
What it does
How often and how quickly a failed record is redelivered before the recoverer is called.
When to change it
Replace it with an exponential backoff, and declare exceptions that can never succeed as not retryable so they go to the dead letter topic at once.
saga deadlineDefault none (application setting, 15 minutes in this lesson)
What it does
How long an order may stay PENDING before the sweeper cancels it.
When to change it
Well above the normal completion time plus the retry budget of every step, and below what a customer will tolerate. Too short and healthy sagas are cancelled while still in flight.
provider idempotency keyDefault none (application setting, the order id in this lesson)
What it does
The key sent with every authorise and refund call so the provider can recognise a repeat.
When to change it
Derive it from the saga, never generate it per attempt. A fresh key on each retry turns a redelivery into a second charge.

Hands-on exercise

Find the double refund

See why a saga step must be idempotent by crashing a payment step between its effect and its offset commit, then decide which defence each failure needs.

Steps · 0 of 5 done

A payment step that crashes before its offset is committed

Simulation · not a real cluster

Run the crash with the idempotent consumer off, then on, and watch how many times the payment effect is applied.

Take it further in the lab: Duplicate message processing

What can go wrong

Compensation that is not idempotent
Symptom
A customer receives two refunds for one cancelled order, or stock for one order is added back twice and the count drifts upward.
Cause
The compensation is triggered by more than one event (rejection and timeout) or by a redelivery, and it applies its effect unconditionally.
Fix
Guard every compensation on the current state (WHERE status = 'AUTHORIZED'), deduplicate by event id, and pass a saga-derived idempotency key to external systems.
Sagas that never finish
Symptom
Orders sit in PENDING for hours. Money is held on customers' cards for orders that will never ship.
Cause
A participant is down or lagging, or its event went to a dead letter topic, and nothing in a choreographed saga is waiting for it.
Fix
A deadline with a sweeper that cancels and publishes the cancellation, an alert on the count of orders pending past the deadline, and an alert on every saga dead letter topic.
Assuming order across topics
Symptom
Inventory occasionally fails with 'order not found' when handling a payment event, mostly under load or after a restart.
Cause
OrderCreated and PaymentAuthorized are on different topics. Kafka orders records within one partition only, and the two partitions are consumed independently.
Fix
Handle the missing predecessor explicitly: fail and retry with backoff, or park the event. Or carry the needed data in the later event so the step does not depend on the earlier one.
Publishing the event outside the step's transaction
Symptom
A payment is authorised but the order never moves on, or a StockReserved event exists for a reservation that was rolled back.
Cause
The listener updates its database and calls KafkaTemplate.send directly. This is the dual write from the outbox lesson, now in the middle of a saga where a lost event stalls it and a phantom event corrupts it.
Fix
Write the outgoing event to the outbox in the same transaction as the state change, in every participant.
Acting on intermediate state
Symptom
The warehouse starts picking an order that is cancelled a minute later. A report counts revenue for orders that were refunded.
Cause
Another process read state written by a saga step that was later compensated. A saga has no isolation, so that state was really there.
Fix
Expose the pending status and make downstream processes wait for the final event (OrderConfirmed). Use reservations instead of final effects until the saga completes.
An irreversible step in the middle
Symptom
Customers get an 'order confirmed' email followed by a cancellation, or a parcel is dispatched for a payment that then fails.
Cause
A step that cannot be compensated runs before steps that can still fail.
Fix
Reorder: every step that can fail for business reasons comes first, irreversible steps come after the last one that can fail, and they retry until they succeed.

How to troubleshoot

One order is stuck in PENDING

Check

kafka-console-consumer.sh --bootstrap-server broker:9092 --topic payments.events --from-beginning --property print.key=true --property print.headers=true | grep o-1001
How to read it
Repeat for each of the three topics to find the last event published for that order id. The saga stopped at the service that should have reacted to it. On large topics start from a timestamp instead of the beginning. Recent 4.x releases deprecate --property here in favour of --formatter-property.
Many orders are stuck at the same step

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group inventory-service
How to read it
Growing lag on payments.events for the group means Inventory is not keeping up or is stopped. Lag stuck on one partition with the same CURRENT-OFFSET suggests a record that is being retried over and over.
A step failed permanently

Check

kafka-console-consumer.sh --bootstrap-server broker:9092 --topic inventory.events-dlt --from-beginning --property print.headers=true
How to read it
Each record in the dead letter topic is one saga that cannot continue. The exception headers added by the recoverer say why. Fix the cause and republish, or complete the compensation by hand.
Events seem not to be published at all

Check

SELECT aggregatetype, count(*), min(seq) FROM outbox_event WHERE published_at IS NULL GROUP BY aggregatetype;
How to read it
Rows waiting in a participant's outbox mean its relay is down. The step committed and the saga will continue when the relay publishes. An empty outbox with a missing event means the step itself never ran.
How many sagas are unfinished right now

Check

SELECT count(*), min(created_at) FROM orders WHERE status = 'PENDING';
How to read it
A small count of recent orders is normal work in progress. Any order older than the deadline means the sweeper is not running or its cancellation is failing.

Key takeaways

  • A saga replaces one distributed transaction with a sequence of local transactions and a compensating action for each step that may need undoing.
  • Compensations are business operations with visible effects. They must be idempotent and retried until they succeed.
  • Choreography spreads the flow across subscriptions and suits short flows. Orchestration puts the flow in one coordinator and suits long or branching ones.
  • Every step: deduplicate the event, check and change state under a guard, write the next event to the outbox, all in one local transaction.
  • Sagas have no isolation. Use pending states, reservations and write-time guards, and put irreversible steps last.
  • Without a timeout a choreographed saga can stay unfinished forever. Count the pending-too-long orders and alert on them.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

Inventory was down for 40 minutes. Afterwards finance finds 23 orders that were refunded twice. For each, the Payment Service log shows a refund on OrderCancelled (reason TIMEOUT) and another on StockRejected a few minutes later. The refund handler deduplicates by event id and then calls the provider with a newly generated request id.

Which change prevents the second refund?

Scenario 2 of 3

A checkout flow has grown to nine steps across six services, with a fraud review that can take up to two days and three different cancellation paths. It is choreographed. On-call engineers regularly need an hour to find out why a given order is not progressing, and adding a step last month required changes in four services.

What is the most appropriate direction?

Scenario 3 of 3

A warehouse service subscribes to payments.events and starts picking as soon as it sees PaymentAuthorized, to save time. About 2% of picked orders are later cancelled because stock was rejected for another line or the fraud check failed, and staff have to put items back.

What is wrong with the design?

References

References: Apache Kafka 4.1: consumer configuration reference (enable.auto.commit, max.poll.interval.ms, auto.offset.reset) ↗ · Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler, back-off, dead letter publishing) ↗ · Spring for Apache Kafka 3.3 reference: Message listener containers (AckMode, enable.auto.commit) ↗ · Spring for Apache Kafka 3.3 reference: Transactions (a database transaction on a listener method) ↗ · Hector Garcia-Molina and Kenneth Salem, 'Sagas', ACM SIGMOD 1987 (the original paper) ↗ · Chris Richardson, microservices.io: Saga pattern (choreography and orchestration) ↗ · Report a problem