Path 1 · lesson 3 of 6
Event-driven architecture: sync calls vs async events
Events, commands and queries are different messages with different coupling. See what moving an order flow from synchronous calls to events buys, and what it costs.
What you will learn
- Classify a message as an event, a command or a query, and say who owns its meaning
- Trace how a failure and a slow dependency spread through a synchronous call chain and through an event flow
- Sketch the order, payment, inventory and notification flow as a choreography of events on Kafka topics
- Name the costs of going asynchronous (eventual consistency, duplicates, harder debugging) and the standard answer to each
- Decide which steps of a workflow should stay synchronous
Before you start
The concept
Services send each other three different kinds of message. They are easy to confuse because all three can travel over HTTP or over Kafka, but they mean different things and create different dependencies.
| Event | Command | Query | |
|---|---|---|---|
| What it says | Something happened | Do this | Tell me this |
| Name | Past tense: OrderPlaced, PaymentAuthorized | Imperative: ChargePayment, ReserveStock | A question: GetOrderStatus |
| Addressed to | Nobody in particular. Anyone may listen | One specific receiver | One specific owner of the data |
| Can it be refused? | No. It is a fact about the past | Yes. The receiver may reject it | It can fail, but it changes nothing |
| Sender expects | Nothing | The action to be carried out, and often a result | An answer, now |
| Who knows whom | Consumers know the producer's event. The producer knows no consumer | The sender knows the receiver and what it can do | The asker knows the owner |
The direction of knowledge in the last row is the heart of event-driven architecture. With a command, the order service must know that a payment service exists and what to ask of it. With an event, the order service announces OrderPlaced and its job ends there. Payment, inventory and whoever arrives next year decide for themselves what that fact means for them.
An event-driven architecture is one where services mostly cooperate by publishing events and reacting to them, instead of calling each other and waiting. Kafka suits it because a topic keeps events in order, lets many services read them independently, and holds them while a service is down.
Synchronous and asynchronous
Synchronous request and response
- The caller waits for the answer before it continues
- Both sides must be up at the same moment (coupled in time)
- The caller's latency includes the latency of everything it calls
- A failure downstream becomes a failure, or a timeout, upstream
- The result is known immediately, so the user can be told yes or no
- The flow is visible in one place: read the calling code
Asynchronous events
- The producer records the fact and continues without waiting
- The consumer can be down; the event waits in the log
- The producer's latency is its own work plus one write to Kafka
- A failing consumer falls behind; the producer is unaffected
- The overall result arrives later, so state is consistent eventually, not instantly
- The flow is spread over several services and has to be reconstructed
Why it exists
Take the running example: a customer places an order, the card is charged, stock is reserved and a confirmation is sent. The first version of most systems does this with calls, one after another, inside the checkout request.
Drawing diagram…
The customer calls the order service, which calls the payment, inventory and notification services in turn and waits for each. The notification service in turn waits for an external email provider.
flowchart LR C["Customer"] -->|"POST /orders and wait"| O["Order service"] O -->|"1 charge and wait"| P["Payment service"] O -->|"2 reserve and wait"| I["Inventory service"] O -->|"3 send and wait"| N["Notification service"] N -->|"SMTP and wait"| M["Email provider"]
This is simple to write and to read, and it has three structural problems.
- Latency adds up. The customer waits for the sum of all four steps, including the email provider's response time, which has nothing to do with whether the order is valid.
- Failures propagate upward. If the email provider is down, the notification call fails or times out, and the order service has to decide inside the request what to do. Often the whole checkout fails because a confirmation email could not be sent. As an illustration of the arithmetic: a request that needs four services which are each up 99.9% of the time can succeed at most about 99.6% of the time.
- Every new step edits the order service. Adding loyalty points or fraud scoring means another call, another timeout and another failure case in the checkout code, owned by a team that does not own those features.
Partial failure is worse still. If payment succeeded and the inventory call timed out, the order service does not know whether stock was reserved. It must retry, or undo the charge, from inside a request whose caller may have gone away.
The event-driven version changes the shape. The order service saves the order as PENDING, publishes OrderPlaced and answers the customer. Each later step is a separate service reacting to an event and publishing its own. The email provider being down now delays emails and nothing else. A slow payment gateway makes orders take longer to confirm, but checkout stays fast. A new consumer is added without touching checkout.
How it works inside
Choreography: each service reacts and announces
In a choreography there is no coordinator. Each service subscribes to the events it cares about, does its own work in its own database, and publishes an event about the outcome. The workflow is the chain of those reactions.
- The order service stores the order as
PENDINGand publishesOrderPlacedtoorders. - The payment service consumes
OrderPlaced, charges the card and publishesPaymentAuthorizedorPaymentDeclinedtopayments. - The inventory service consumes
PaymentAuthorized, reserves stock and publishesStockReservedorStockRejectedtoinventory. - The order service consumes those outcomes and moves the order to
CONFIRMEDorCANCELLED. - The notification service consumes the outcomes and tells the customer.
Failure paths are events as well. If stock is rejected after payment succeeded, the payment service must react to StockRejected by refunding. Such undo steps are called compensations, and a workflow built from local steps and compensations is a saga. The alternative to choreography is orchestration, where one component sends commands and tracks the state of each order. The saga pattern lesson compares the two.
What Kafka contributes
- Time decoupling. Events are retained, so a consumer that is down for twenty minutes finds them waiting. Its lag grows and then shrinks. No upstream service retries or buffers on its behalf.
- Order per entity. Every event is keyed by order id, so all events for one order in a topic are in one partition, in the order they were appended. Order between different topics, for example
ordersandpayments, is not guaranteed, so consumers must not assume it. - Independent readers. Each service has its own consumer group and therefore its own offsets. A new service gets its own group and can read the retained history.
- At-least-once delivery by default. A consumer that processes a record and crashes before its offset is committed will receive that record again after restart or rebalance. This is the source of duplicates in an event-driven system.
What you now have to handle
| Cost | What it looks like | Standard answer |
|---|---|---|
| Eventual consistency | The customer sees PENDING for a moment. A read just after a write can show old state | Model the in-between states explicitly and show them. Keep steps that must give an immediate yes or no synchronous |
| Duplicates | The same OrderPlaced is delivered twice after a consumer restart | Idempotent consumers: handling an event twice has the same effect as once. See Duplicates and idempotency |
| Harder debugging | "Why was order 1042 never confirmed?" has no single stack trace to read | A correlation id on every event, consumer lag monitoring, and distributed tracing |
| The dual write | The order is saved but the service crashes before publishing, or the reverse | Write the event to an outbox table in the same database transaction. See Transactional outbox |
| Event contracts | A renamed field in OrderPlaced breaks four consumers the producer has never heard of | Treat event schemas as public APIs and evolve them compatibly. See Schema Registry and evolution |
| Hidden flow | Nobody can say what happens after an order is placed without reading five services | Document the event flow, or move complex workflows to an orchestrated saga |
Architecture
Drawing diagram…
The order service publishes OrderPlaced to the orders topic. The payment service consumes it and publishes its outcome to the payments topic. The inventory service consumes payments and publishes to the inventory topic. The order service consumes payments and inventory outcomes, the payment service consumes inventory outcomes for refunds, and the notification service consumes all three topics.
flowchart LR OS["Order service"] -->|"OrderPlaced"| TO["Topic orders"] TO --> PS["Payment service"] PS -->|"PaymentAuthorized or PaymentDeclined"| TP["Topic payments"] TP --> IS["Inventory service"] IS -->|"StockReserved or StockRejected"| TI["Topic inventory"] TP --> OS TI --> OS TI --> PS TO --> NS["Notification service"] TP --> NS TI --> NS
- No arrow goes from one service to another. Every service knows the topics and event types it uses and nothing about who else reads them.
- Each service is in its own consumer group, so each receives every event on the topics it subscribes to and keeps its own offsets.
- The arrow from
inventoryback to the payment service is the compensation path: aStockRejectedafter a successful charge triggers a refund. - All three topics are keyed by order id. Events for one order stay ordered inside each topic, and different orders are processed in parallel across partitions.
Message flow
Drawing diagram…
The customer posts an order. The order service saves it as pending, publishes OrderPlaced and replies 202. The payment service consumes the event, charges the card and publishes PaymentAuthorized. The inventory service consumes that, reserves stock and publishes StockReserved. The order service marks the order confirmed and the notification service emails the customer.
sequenceDiagram participant C as Customer participant O as Order service participant K as Kafka participant P as Payment service participant I as Inventory service participant N as Notification service C->>O: POST /orders O->>O: Save order as PENDING O->>K: OrderPlaced to orders O-->>C: 202 Accepted - order pending K-->>P: OrderPlaced P->>P: Charge card P->>K: PaymentAuthorized to payments K-->>I: PaymentAuthorized I->>I: Reserve stock I->>K: StockReserved to inventory K-->>O: StockReserved O->>O: Mark order CONFIRMED K-->>N: StockReserved N->>C: Confirmation email
- The response to the customer does not wait for payment, stock or email. That is where the latency improvement comes from, and also why the response can only say "pending".
- Each dashed arrow from Kafka is a consumer pulling the event when it is ready. If the inventory service is down at that moment, the arrow simply happens later.
- Between the 202 and
CONFIRMEDthe system is in an intermediate state on purpose. The user interface has to show it, and the order service has to handle the case where the outcome is a decline or a rejection.
In a real system
A typical layout for this flow on Kafka. It mirrors the academy's example project, which you can read service by service.
| Topic | Events | Key | Produced by | Consumer groups |
|---|---|---|---|---|
orders | OrderPlaced, OrderConfirmed, OrderCancelled | order id | Order service | payment-service, notification-service, analytics |
payments | PaymentAuthorized, PaymentDeclined, PaymentRefunded | order id | Payment service | inventory-service, order-service, notification-service |
inventory | StockReserved, StockRejected | order id | Inventory service | order-service, payment-service, notification-service |
Each topic has, say, 6 partitions and a replication factor of 3. The order id is the key everywhere, so the events of one order are ordered within each topic and up to six instances of a service can work in parallel. Every event carries a unique event id, the order id as a correlation id, and a timestamp.
What stays synchronous
Not everything should become an event. The product page asks the catalogue for a price and needs the answer now: that is a query, and a request and response fits it. Checking that the cart is not empty is validation inside the order service. And if the business requires the customer to see "payment declined" on the checkout page itself, the charge may stay a synchronous command while stock, email and analytics hang off events. A good default: keep on the request path only what the user must know before the page can respond, and move the rest behind an event.
Now consider twenty minutes of email provider outage in both designs. In the synchronous chain, checkout fails or hangs for twenty minutes. In the event-driven design the notification-service group's lag grows for twenty minutes, orders keep being placed, paid and confirmed, and the emails go out when the provider returns.
Java and Spring Boot
One step of the choreography: the payment service reacts to OrderPlaced and publishes the outcome. It is written so that receiving the same event twice does not charge the customer twice.
package com.example.payments.events;/** Published by the order service. The payment service owns this copy of the contract. */public record OrderPlaced(String eventId, String orderId, String customerId, long amountMinor, String currency) {}Why it is written this way
eventIdidentifies this event;orderIdidentifies the order and doubles as the correlation id that ties the whole flow together in logs.- The consumer keeps its own record type with the fields it needs. It should not share a Java class with the producer: the contract is the JSON, not the class.
package com.example.payments.events;/** Published by the payment service: the outcome of trying to pay for one order. */public record PaymentResult(String eventId, String orderId, Status status, String reason) { public enum Status { AUTHORIZED, DECLINED } public static PaymentResult authorized(String eventId, String orderId) { return new PaymentResult(eventId, orderId, Status.AUTHORIZED, null); } public static PaymentResult declined(String eventId, String orderId, String reason) { return new PaymentResult(eventId, orderId, Status.DECLINED, reason); }}Why it is written this way
- A decline is a normal business outcome, so it is an event like any other and not an exception.
- One record type per topic keeps JSON deserialization simple for a first example. Several event types on one topic need a type discriminator, covered in Serialization in Spring.
package com.example.payments;/** The card processor, behind an interface. */public interface PaymentGateway { /** * Charges once per idempotency key. Calling it again with the same key must return * the outcome of the first call and must not charge again. */ Outcome charge(String idempotencyKey, long amountMinor, String currency); record Outcome(boolean approved, String declineReason) { }}Why it is written this way
- The idempotency key is what makes a repeated delivery harmless. Most card processors accept one; if yours does not, record handled order ids in your own database first.
package com.example.payments;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Component;import com.example.payments.events.OrderPlaced;import com.example.payments.events.PaymentResult;@Componentpublic class OrderPlacedListener { private static final Logger log = LoggerFactory.getLogger(OrderPlacedListener.class); private final PaymentGateway gateway; private final KafkaTemplate<String, PaymentResult> kafkaTemplate; public OrderPlacedListener(PaymentGateway gateway, KafkaTemplate<String, PaymentResult> kafkaTemplate) { this.gateway = gateway; this.kafkaTemplate = kafkaTemplate; } @KafkaListener(topics = "orders", groupId = "payment-service") public void onOrderPlaced(OrderPlaced event) { // The order id is the idempotency key: a redelivered event cannot charge twice. PaymentGateway.Outcome outcome = gateway.charge(event.orderId(), event.amountMinor(), event.currency()); // Derived from the order id, so a redelivery publishes a result with the same event id. String resultId = "payment-result-" + event.orderId(); PaymentResult result = outcome.approved() ? PaymentResult.authorized(resultId, event.orderId()) : PaymentResult.declined(resultId, event.orderId(), outcome.declineReason()); // Wait for Kafka to confirm before returning. If this throws, the listener fails, // the offset is not committed and the event is delivered again. kafkaTemplate.send("payments", event.orderId(), result).join(); log.info("order {} payment {}", event.orderId(), result.status()); }}Why it is written this way
- The listener returns normally only after the outcome event is stored. Spring's container commits the offset after that, so a crash anywhere before it leads to redelivery, not to a lost payment step. This is at-least-once processing.
- Redelivery is safe because the gateway is called with the same idempotency key and returns the first outcome. This is consumer-side idempotency, built by the application. The producer setting
enable.idempotenceis something else: it only removes duplicates caused by the producer's own retries. - A redelivery can still publish the
PaymentResulta second time. It carries the same event id, so downstream consumers can recognise it, and they must be idempotent as well. join()blocks until the send succeeds or fails; the producer'sdelivery.timeout.ms(2 minutes by default) bounds how long that can take.- The outcome is keyed by order id, like the incoming event, so the order's events stay in one partition of
payments. - If the listener keeps throwing, Spring's default error handler retries the record and then, after ten failed attempts, logs it and moves on. Error handling in Spring shows how to send it to a dead letter topic instead.
spring: kafka: bootstrap-servers: localhost:9092 producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: "[spring.json.add.type.headers]": false consumer: group-id: payment-service auto-offset-reset: earliest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: "[spring.json.value.default.type]": com.example.payments.events.OrderPlaced "[spring.json.trusted.packages]": com.example.payments.eventsWhy it is written this way
auto-offset-reset: earliestmeans that the very first deployment of the payment service processes orders that were placed before it started, instead of silently skipping them.- The consumer maps the JSON onto its own
OrderPlacedclass and ignores type headers, so the two services do not need to share a package name.
Configuration
The settings that decide how an event-driven service behaves when something is slow or down. Kafka defaults are from the Apache Kafka 4.4 documentation; Spring defaults from the Spring for Apache Kafka 3.3 reference.
spring.kafka.listener.ack-modeDefault BATCH
- What it does
- When Spring's listener container commits offsets.
BATCHcommits after all records returned by one poll have been processed by the listener. - When to change it
- The default gives at-least-once processing, which is what an event flow normally wants. Change it only with a specific reason; see @KafkaListener in depth.
enable.auto.commitDefault true (Spring for Apache Kafka sets it to false unless you set it)
- What it does
- Whether the Kafka client commits offsets in the background on a timer, independently of whether your code has finished processing.
- When to change it
- Leave it to Spring. With timer-based commits an offset can be committed for an event whose handler then fails, and that event is skipped.
auto.offset.resetDefault latest
- What it does
- Where a consumer group with no committed offset begins reading.
- When to change it
- For a service that must act on every event, set
earliest, or events published before its first start are never processed.
max.poll.interval.msDefault 300000 (5 minutes)
- What it does
- The longest allowed gap between two polls. If a handler takes longer, the consumer is considered failed and its partitions are given to another member.
- When to change it
- A handler that calls a slow external system must finish a whole poll's worth of records (up to
max.poll.records, 500 by default) within this time. Put timeouts on outbound calls before raising it.
max.block.msDefault 60000 (1 minute)
- What it does
- How long the producer's
send()may block while waiting for metadata or buffer space, for example when no broker is reachable. - When to change it
- If a request thread publishes directly, this is how long a user can wait during a Kafka outage. Lower it for request paths, or publish from an outbox instead.
delivery.timeout.msDefault 120000 (2 minutes)
- What it does
- The upper bound on the time between
send()returning and the producer reporting success or failure, including retries. - When to change it
- Until the returned future completes you do not know the event was stored. Decide what the service does when it completes with an error.
Hands-on exercise
Produce a duplicate, then make it harmless
See where duplicates come from in an event-driven flow and why the fix belongs in the consumer.
Steps · 0 of 6 done
A payment consumer that crashes between work and commit
Simulation · not a real clusterRun the scenario, read what happened to the customer's card, then change the crash point and the idempotent consumer option.
What you should see
With commit after processing and a crash in between, the restarted consumer resumes from the uncommitted offset and charges a second time. With the idempotent consumer on, the event is still delivered twice but the second delivery changes nothing. With commit before processing, the crash loses the payment entirely: the offset has moved on and the event is never handled.
What can go wrong
Dual write: database and Kafka disagree
- Symptom
- Orders exist in the database with status
PENDINGforever and noOrderPlacedevent; or payments are taken for orders that were rolled back. - Cause
- The service commits to its database and then calls
send(or the reverse). A crash, or a failed send, between the two leaves one done and the other not. No Kafka setting joins the two writes. - Fix
- Write the event into an outbox table in the same database transaction as the order, and publish from that table. See Transactional outbox.
Handlers that are not idempotent
- Symptom
- After a deployment or a rebalance, a few customers are charged twice or stock is decremented twice.
- Cause
- Delivery is at-least-once: records processed after the last committed offset are delivered again. The handler repeats its side effect.
- Fix
- Make each handler safe to repeat, with an idempotency key on external calls or a processed-event table updated in the same transaction as the business change.
Reading your own write too early
- Symptom
- The customer is redirected to the order page right after checkout and sees "order not found" or a stale status.
- Cause
- The page reads from a view that is updated by consuming events, and the event has not been consumed yet.
- Fix
- Serve the immediate read from the service that accepted the write, show the pending state explicitly, and let the page update when the outcome arrives.
Assuming order across topics
- Symptom
- The order service occasionally receives
StockReservedfor an order before it has seenPaymentAuthorized. - Cause
- Ordering holds within one partition of one topic. Two topics are consumed independently, at different speeds.
- Fix
- Make handlers tolerate either arrival order, for example by keeping per-order state and acting when both facts are present, or put events that must be ordered on one topic with one key.
A failing event is skipped quietly
- Symptom
- An order never progresses. There is no lag and no alert; one ERROR line in the consumer log from hours ago mentions its offset.
- Cause
- With Spring's default error handler, a record that keeps failing is retried, then logged and skipped after ten failed attempts, so that the partition is not blocked.
- Fix
- Configure a recoverer that publishes failed records to a dead letter topic, and alert on that topic. See Dead letter queues.
A distributed monolith with extra steps
- Symptom
- Services publish events and then immediately call each other over HTTP for the details, so an outage of one still stops the rest.
- Cause
- Events carry only ids, or are used as triggers for synchronous calls, so the coupling in time was never removed.
- Fix
- Put the data consumers need in the event, or let consumers keep their own copy of it built from events.
How to troubleshoot
An order is stuck in PENDING
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- Find which step stopped by checking each service's group in flow order. Growing
LAGon a group means that service is down or slow and the event is still waiting. Zero lag everywhere means the event was consumed, so look for it in that service's logs by order id.
Need to confirm an event was actually published
Check
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning | grep order-1042
- How to read it
- Reads the topic independently of every service's group. If the event is absent, the problem is on the producing side (a failed send or a dual write). If present, the problem is in a consumer.
Duplicate side effects after a deployment
Check
The consumer's log for partition revocation and assignment around the time of the duplicates, and the handler for a side effect that is not idempotent
- How to read it
- A restart or rebalance hands partitions to a consumer that resumes from the last committed offset. Anything processed after that commit runs again. This is expected behaviour, so the handler is what needs fixing.
Checkout hangs when Kafka is unhealthy
Check
The order service log for `TimeoutException` from the producer, and the value of `max.block.ms`
- How to read it
- A send on the request thread waits for broker metadata for up to
max.block.ms. The service is asynchronous towards other services and still synchronous towards Kafka.
Key takeaways
- An event states a fact and is addressed to no one. A command asks one receiver to act. A query asks for data. Name each honestly.
- Synchronous calls couple services in time: latencies add up and a downstream failure becomes an upstream failure.
- With events the producer finishes after one write to Kafka. A slow or failed consumer falls behind and catches up, and new consumers need no change to the producer.
- In a choreography each service reacts to events and publishes its own, including failure outcomes that trigger compensations.
- The price is eventual consistency, duplicate deliveries and a flow that is harder to see. Plan for pending states, idempotent handlers and a correlation id from the start.
- Saving to a database and publishing to Kafka are two separate writes. Use an outbox when they must agree.
- Keep synchronous what the user must know before the response. Move the rest behind events.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
Checkout calls payment, then inventory, then the notification service, all synchronously. Last night the email provider was unreachable for 25 minutes. During that time 100% of checkouts failed with a 504, although payment and inventory were healthy.
Which change removes this failure mode with the least risk to the order flow?
Scenario 2 of 3
The inventory service consumes PaymentAuthorized and decrements stock. After each rolling deployment, stock for a handful of products is 1 or 2 lower than it should be. Producer logs show each PaymentAuthorized was sent once. The listener commits offsets after processing.
What should the team change?
Scenario 3 of 3
The product page must show the current price. A developer proposes that the web tier publish a PriceRequested event with a request id to Kafka, have the pricing service publish PriceProvided, and wait up to 2 seconds for the matching reply before rendering.
What is the better design for this interaction?
References
References: Apache Kafka documentation: Introduction (events, producers and consumers are decoupled) ↗ · Apache Kafka 4.4 documentation: Design (message delivery semantics) ↗ · Apache Kafka 4.4 documentation: Producer configs ↗ · Apache Kafka 4.4 documentation: Consumer configs ↗ · Spring for Apache Kafka 3.3 reference: Message listener containers (committing offsets, AckMode) ↗ · Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler) ↗ · Spring for Apache Kafka 3.3 reference: Sending messages ↗ · Martin Fowler: What do you mean by "Event-Driven"? ↗ · Report a problem