Skip to content

Path 1 · lesson 3 of 6

Foundation22 minApache Kafka 4.4 (KRaft) · Java 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 (kafka-clients 3.9)

Event-driven architecture: sync calls vs async events

Events, commands and queries are different messages with different coupling. See what moving an order flow from synchronous calls to events buys, and what it costs.

What you will learn

  • Classify a message as an event, a command or a query, and say who owns its meaning
  • Trace how a failure and a slow dependency spread through a synchronous call chain and through an event flow
  • Sketch the order, payment, inventory and notification flow as a choreography of events on Kafka topics
  • Name the costs of going asynchronous (eventual consistency, duplicates, harder debugging) and the standard answer to each
  • Decide which steps of a workflow should stay synchronous

Before you start

The concept

Services send each other three different kinds of message. They are easy to confuse because all three can travel over HTTP or over Kafka, but they mean different things and create different dependencies.

EventCommandQuery
What it saysSomething happenedDo thisTell me this
NamePast tense: OrderPlaced, PaymentAuthorizedImperative: ChargePayment, ReserveStockA question: GetOrderStatus
Addressed toNobody in particular. Anyone may listenOne specific receiverOne specific owner of the data
Can it be refused?No. It is a fact about the pastYes. The receiver may reject itIt can fail, but it changes nothing
Sender expectsNothingThe action to be carried out, and often a resultAn answer, now
Who knows whomConsumers know the producer's event. The producer knows no consumerThe sender knows the receiver and what it can doThe asker knows the owner

The direction of knowledge in the last row is the heart of event-driven architecture. With a command, the order service must know that a payment service exists and what to ask of it. With an event, the order service announces OrderPlaced and its job ends there. Payment, inventory and whoever arrives next year decide for themselves what that fact means for them.

An event-driven architecture is one where services mostly cooperate by publishing events and reacting to them, instead of calling each other and waiting. Kafka suits it because a topic keeps events in order, lets many services read them independently, and holds them while a service is down.

Synchronous and asynchronous

Synchronous request and response

  • The caller waits for the answer before it continues
  • Both sides must be up at the same moment (coupled in time)
  • The caller's latency includes the latency of everything it calls
  • A failure downstream becomes a failure, or a timeout, upstream
  • The result is known immediately, so the user can be told yes or no
  • The flow is visible in one place: read the calling code

Asynchronous events

  • The producer records the fact and continues without waiting
  • The consumer can be down; the event waits in the log
  • The producer's latency is its own work plus one write to Kafka
  • A failing consumer falls behind; the producer is unaffected
  • The overall result arrives later, so state is consistent eventually, not instantly
  • The flow is spread over several services and has to be reconstructed

Why it exists

Take the running example: a customer places an order, the card is charged, stock is reserved and a confirmation is sent. The first version of most systems does this with calls, one after another, inside the checkout request.

Drawing diagram…

The customer calls the order service, which calls the payment, inventory and notification services in turn and waits for each. The notification service in turn waits for an external email provider.

flowchart LR
  C["Customer"] -->|"POST /orders and wait"| O["Order service"]
  O -->|"1 charge and wait"| P["Payment service"]
  O -->|"2 reserve and wait"| I["Inventory service"]
  O -->|"3 send and wait"| N["Notification service"]
  N -->|"SMTP and wait"| M["Email provider"]
The synchronous version: checkout is as slow and as available as the whole chain behind it.

This is simple to write and to read, and it has three structural problems.

  • Latency adds up. The customer waits for the sum of all four steps, including the email provider's response time, which has nothing to do with whether the order is valid.
  • Failures propagate upward. If the email provider is down, the notification call fails or times out, and the order service has to decide inside the request what to do. Often the whole checkout fails because a confirmation email could not be sent. As an illustration of the arithmetic: a request that needs four services which are each up 99.9% of the time can succeed at most about 99.6% of the time.
  • Every new step edits the order service. Adding loyalty points or fraud scoring means another call, another timeout and another failure case in the checkout code, owned by a team that does not own those features.

Partial failure is worse still. If payment succeeded and the inventory call timed out, the order service does not know whether stock was reserved. It must retry, or undo the charge, from inside a request whose caller may have gone away.

The event-driven version changes the shape. The order service saves the order as PENDING, publishes OrderPlaced and answers the customer. Each later step is a separate service reacting to an event and publishing its own. The email provider being down now delays emails and nothing else. A slow payment gateway makes orders take longer to confirm, but checkout stays fast. A new consumer is added without touching checkout.

How it works inside

Choreography: each service reacts and announces

In a choreography there is no coordinator. Each service subscribes to the events it cares about, does its own work in its own database, and publishes an event about the outcome. The workflow is the chain of those reactions.

  1. The order service stores the order as PENDING and publishes OrderPlaced to orders.
  2. The payment service consumes OrderPlaced, charges the card and publishes PaymentAuthorized or PaymentDeclined to payments.
  3. The inventory service consumes PaymentAuthorized, reserves stock and publishes StockReserved or StockRejected to inventory.
  4. The order service consumes those outcomes and moves the order to CONFIRMED or CANCELLED.
  5. The notification service consumes the outcomes and tells the customer.

Failure paths are events as well. If stock is rejected after payment succeeded, the payment service must react to StockRejected by refunding. Such undo steps are called compensations, and a workflow built from local steps and compensations is a saga. The alternative to choreography is orchestration, where one component sends commands and tracks the state of each order. The saga pattern lesson compares the two.

What Kafka contributes

  • Time decoupling. Events are retained, so a consumer that is down for twenty minutes finds them waiting. Its lag grows and then shrinks. No upstream service retries or buffers on its behalf.
  • Order per entity. Every event is keyed by order id, so all events for one order in a topic are in one partition, in the order they were appended. Order between different topics, for example orders and payments, is not guaranteed, so consumers must not assume it.
  • Independent readers. Each service has its own consumer group and therefore its own offsets. A new service gets its own group and can read the retained history.
  • At-least-once delivery by default. A consumer that processes a record and crashes before its offset is committed will receive that record again after restart or rebalance. This is the source of duplicates in an event-driven system.

What you now have to handle

CostWhat it looks likeStandard answer
Eventual consistencyThe customer sees PENDING for a moment. A read just after a write can show old stateModel the in-between states explicitly and show them. Keep steps that must give an immediate yes or no synchronous
DuplicatesThe same OrderPlaced is delivered twice after a consumer restartIdempotent consumers: handling an event twice has the same effect as once. See Duplicates and idempotency
Harder debugging"Why was order 1042 never confirmed?" has no single stack trace to readA correlation id on every event, consumer lag monitoring, and distributed tracing
The dual writeThe order is saved but the service crashes before publishing, or the reverseWrite the event to an outbox table in the same database transaction. See Transactional outbox
Event contractsA renamed field in OrderPlaced breaks four consumers the producer has never heard ofTreat event schemas as public APIs and evolve them compatibly. See Schema Registry and evolution
Hidden flowNobody can say what happens after an order is placed without reading five servicesDocument the event flow, or move complex workflows to an orchestrated saga

Architecture

Drawing diagram…

The order service publishes OrderPlaced to the orders topic. The payment service consumes it and publishes its outcome to the payments topic. The inventory service consumes payments and publishes to the inventory topic. The order service consumes payments and inventory outcomes, the payment service consumes inventory outcomes for refunds, and the notification service consumes all three topics.

flowchart LR
  OS["Order service"] -->|"OrderPlaced"| TO["Topic orders"]
  TO --> PS["Payment service"]
  PS -->|"PaymentAuthorized or PaymentDeclined"| TP["Topic payments"]
  TP --> IS["Inventory service"]
  IS -->|"StockReserved or StockRejected"| TI["Topic inventory"]
  TP --> OS
  TI --> OS
  TI --> PS
  TO --> NS["Notification service"]
  TP --> NS
  TI --> NS
The order flow as a choreography: services only publish to topics and consume from topics.
  • No arrow goes from one service to another. Every service knows the topics and event types it uses and nothing about who else reads them.
  • Each service is in its own consumer group, so each receives every event on the topics it subscribes to and keeps its own offsets.
  • The arrow from inventory back to the payment service is the compensation path: a StockRejected after a successful charge triggers a refund.
  • All three topics are keyed by order id. Events for one order stay ordered inside each topic, and different orders are processed in parallel across partitions.

Message flow

Drawing diagram…

The customer posts an order. The order service saves it as pending, publishes OrderPlaced and replies 202. The payment service consumes the event, charges the card and publishes PaymentAuthorized. The inventory service consumes that, reserves stock and publishes StockReserved. The order service marks the order confirmed and the notification service emails the customer.

sequenceDiagram
  participant C as Customer
  participant O as Order service
  participant K as Kafka
  participant P as Payment service
  participant I as Inventory service
  participant N as Notification service
  C->>O: POST /orders
  O->>O: Save order as PENDING
  O->>K: OrderPlaced to orders
  O-->>C: 202 Accepted - order pending
  K-->>P: OrderPlaced
  P->>P: Charge card
  P->>K: PaymentAuthorized to payments
  K-->>I: PaymentAuthorized
  I->>I: Reserve stock
  I->>K: StockReserved to inventory
  K-->>O: StockReserved
  O->>O: Mark order CONFIRMED
  K-->>N: StockReserved
  N->>C: Confirmation email
The happy path. The customer gets an answer after the third step; everything else happens afterwards.
  1. The response to the customer does not wait for payment, stock or email. That is where the latency improvement comes from, and also why the response can only say "pending".
  2. Each dashed arrow from Kafka is a consumer pulling the event when it is ready. If the inventory service is down at that moment, the arrow simply happens later.
  3. Between the 202 and CONFIRMED the system is in an intermediate state on purpose. The user interface has to show it, and the order service has to handle the case where the outcome is a decline or a rejection.

In a real system

A typical layout for this flow on Kafka. It mirrors the academy's example project, which you can read service by service.

TopicEventsKeyProduced byConsumer groups
ordersOrderPlaced, OrderConfirmed, OrderCancelledorder idOrder servicepayment-service, notification-service, analytics
paymentsPaymentAuthorized, PaymentDeclined, PaymentRefundedorder idPayment serviceinventory-service, order-service, notification-service
inventoryStockReserved, StockRejectedorder idInventory serviceorder-service, payment-service, notification-service

Each topic has, say, 6 partitions and a replication factor of 3. The order id is the key everywhere, so the events of one order are ordered within each topic and up to six instances of a service can work in parallel. Every event carries a unique event id, the order id as a correlation id, and a timestamp.

What stays synchronous

Not everything should become an event. The product page asks the catalogue for a price and needs the answer now: that is a query, and a request and response fits it. Checking that the cart is not empty is validation inside the order service. And if the business requires the customer to see "payment declined" on the checkout page itself, the charge may stay a synchronous command while stock, email and analytics hang off events. A good default: keep on the request path only what the user must know before the page can respond, and move the rest behind an event.

Now consider twenty minutes of email provider outage in both designs. In the synchronous chain, checkout fails or hangs for twenty minutes. In the event-driven design the notification-service group's lag grows for twenty minutes, orders keep being placed, paid and confirmed, and the emails go out when the provider returns.

Java and Spring Boot

One step of the choreography: the payment service reacts to OrderPlaced and publishes the outcome. It is written so that receiving the same event twice does not charge the customer twice.

src/main/java/com/example/payments/events/OrderPlaced.javaJava
package com.example.payments.events;/** Published by the order service. The payment service owns this copy of the contract. */public record OrderPlaced(String eventId, String orderId, String customerId, long amountMinor, String currency) {}

Why it is written this way

  • eventId identifies this event; orderId identifies the order and doubles as the correlation id that ties the whole flow together in logs.
  • The consumer keeps its own record type with the fields it needs. It should not share a Java class with the producer: the contract is the JSON, not the class.

Configuration

The settings that decide how an event-driven service behaves when something is slow or down. Kafka defaults are from the Apache Kafka 4.4 documentation; Spring defaults from the Spring for Apache Kafka 3.3 reference.

spring.kafka.listener.ack-modeDefault BATCH
What it does
When Spring's listener container commits offsets. BATCH commits after all records returned by one poll have been processed by the listener.
When to change it
The default gives at-least-once processing, which is what an event flow normally wants. Change it only with a specific reason; see @KafkaListener in depth.
enable.auto.commitDefault true (Spring for Apache Kafka sets it to false unless you set it)
What it does
Whether the Kafka client commits offsets in the background on a timer, independently of whether your code has finished processing.
When to change it
Leave it to Spring. With timer-based commits an offset can be committed for an event whose handler then fails, and that event is skipped.
auto.offset.resetDefault latest
What it does
Where a consumer group with no committed offset begins reading.
When to change it
For a service that must act on every event, set earliest, or events published before its first start are never processed.
max.poll.interval.msDefault 300000 (5 minutes)
What it does
The longest allowed gap between two polls. If a handler takes longer, the consumer is considered failed and its partitions are given to another member.
When to change it
A handler that calls a slow external system must finish a whole poll's worth of records (up to max.poll.records, 500 by default) within this time. Put timeouts on outbound calls before raising it.
max.block.msDefault 60000 (1 minute)
What it does
How long the producer's send() may block while waiting for metadata or buffer space, for example when no broker is reachable.
When to change it
If a request thread publishes directly, this is how long a user can wait during a Kafka outage. Lower it for request paths, or publish from an outbox instead.
delivery.timeout.msDefault 120000 (2 minutes)
What it does
The upper bound on the time between send() returning and the producer reporting success or failure, including retries.
When to change it
Until the returned future completes you do not know the event was stored. Decide what the service does when it completes with an error.

Hands-on exercise

Produce a duplicate, then make it harmless

See where duplicates come from in an event-driven flow and why the fix belongs in the consumer.

Steps · 0 of 6 done

A payment consumer that crashes between work and commit

Simulation · not a real cluster

Run the scenario, read what happened to the customer's card, then change the crash point and the idempotent consumer option.

Take it further in the lab: Duplicate message processing

What can go wrong

Dual write: database and Kafka disagree
Symptom
Orders exist in the database with status PENDING forever and no OrderPlaced event; or payments are taken for orders that were rolled back.
Cause
The service commits to its database and then calls send (or the reverse). A crash, or a failed send, between the two leaves one done and the other not. No Kafka setting joins the two writes.
Fix
Write the event into an outbox table in the same database transaction as the order, and publish from that table. See Transactional outbox.
Handlers that are not idempotent
Symptom
After a deployment or a rebalance, a few customers are charged twice or stock is decremented twice.
Cause
Delivery is at-least-once: records processed after the last committed offset are delivered again. The handler repeats its side effect.
Fix
Make each handler safe to repeat, with an idempotency key on external calls or a processed-event table updated in the same transaction as the business change.
Reading your own write too early
Symptom
The customer is redirected to the order page right after checkout and sees "order not found" or a stale status.
Cause
The page reads from a view that is updated by consuming events, and the event has not been consumed yet.
Fix
Serve the immediate read from the service that accepted the write, show the pending state explicitly, and let the page update when the outcome arrives.
Assuming order across topics
Symptom
The order service occasionally receives StockReserved for an order before it has seen PaymentAuthorized.
Cause
Ordering holds within one partition of one topic. Two topics are consumed independently, at different speeds.
Fix
Make handlers tolerate either arrival order, for example by keeping per-order state and acting when both facts are present, or put events that must be ordered on one topic with one key.
A failing event is skipped quietly
Symptom
An order never progresses. There is no lag and no alert; one ERROR line in the consumer log from hours ago mentions its offset.
Cause
With Spring's default error handler, a record that keeps failing is retried, then logged and skipped after ten failed attempts, so that the partition is not blocked.
Fix
Configure a recoverer that publishes failed records to a dead letter topic, and alert on that topic. See Dead letter queues.
A distributed monolith with extra steps
Symptom
Services publish events and then immediately call each other over HTTP for the details, so an outage of one still stops the rest.
Cause
Events carry only ids, or are used as triggers for synchronous calls, so the coupling in time was never removed.
Fix
Put the data consumers need in the event, or let consumers keep their own copy of it built from events.

How to troubleshoot

An order is stuck in PENDING

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
Find which step stopped by checking each service's group in flow order. Growing LAG on a group means that service is down or slow and the event is still waiting. Zero lag everywhere means the event was consumed, so look for it in that service's logs by order id.
Need to confirm an event was actually published

Check

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning | grep order-1042
How to read it
Reads the topic independently of every service's group. If the event is absent, the problem is on the producing side (a failed send or a dual write). If present, the problem is in a consumer.
Duplicate side effects after a deployment

Check

The consumer's log for partition revocation and assignment around the time of the duplicates, and the handler for a side effect that is not idempotent
How to read it
A restart or rebalance hands partitions to a consumer that resumes from the last committed offset. Anything processed after that commit runs again. This is expected behaviour, so the handler is what needs fixing.
Checkout hangs when Kafka is unhealthy

Check

The order service log for `TimeoutException` from the producer, and the value of `max.block.ms`
How to read it
A send on the request thread waits for broker metadata for up to max.block.ms. The service is asynchronous towards other services and still synchronous towards Kafka.

Key takeaways

  • An event states a fact and is addressed to no one. A command asks one receiver to act. A query asks for data. Name each honestly.
  • Synchronous calls couple services in time: latencies add up and a downstream failure becomes an upstream failure.
  • With events the producer finishes after one write to Kafka. A slow or failed consumer falls behind and catches up, and new consumers need no change to the producer.
  • In a choreography each service reacts to events and publishes its own, including failure outcomes that trigger compensations.
  • The price is eventual consistency, duplicate deliveries and a flow that is harder to see. Plan for pending states, idempotent handlers and a correlation id from the start.
  • Saving to a database and publishing to Kafka are two separate writes. Use an outbox when they must agree.
  • Keep synchronous what the user must know before the response. Move the rest behind events.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

Checkout calls payment, then inventory, then the notification service, all synchronously. Last night the email provider was unreachable for 25 minutes. During that time 100% of checkouts failed with a 504, although payment and inventory were healthy.

Which change removes this failure mode with the least risk to the order flow?

Scenario 2 of 3

The inventory service consumes PaymentAuthorized and decrements stock. After each rolling deployment, stock for a handful of products is 1 or 2 lower than it should be. Producer logs show each PaymentAuthorized was sent once. The listener commits offsets after processing.

What should the team change?

Scenario 3 of 3

The product page must show the current price. A developer proposes that the web tier publish a PriceRequested event with a request id to Kafka, have the pricing service publish PriceProvided, and wait up to 2 seconds for the matching reply before rendering.

What is the better design for this interaction?

References

References: Apache Kafka documentation: Introduction (events, producers and consumers are decoupled) ↗ · Apache Kafka 4.4 documentation: Design (message delivery semantics) ↗ · Apache Kafka 4.4 documentation: Producer configs ↗ · Apache Kafka 4.4 documentation: Consumer configs ↗ · Spring for Apache Kafka 3.3 reference: Message listener containers (committing offsets, AckMode) ↗ · Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler) ↗ · Spring for Apache Kafka 3.3 reference: Sending messages ↗ · Martin Fowler: What do you mean by "Event-Driven"? ↗ · Report a problem