Path 7 · lesson 4 of 8
Event sourcing
Store every change to an order as an event and derive the current state by replaying them. What Kafka does well as the log, where it falls short as the event store, and what teams do instead.
What you will learn
- Model an order as a sequence of events and rebuild its current state by replaying them
- Append events with an expected version so two concurrent commands cannot both succeed on stale state
- Explain what Kafka provides for event sourcing and the two things it lacks, and choose a design that covers them
- Build a projection from the event stream that is idempotent and can be rebuilt from scratch
- Use snapshots and compacted topics correctly, without destroying the history they summarise
- Plan for event schema evolution and for erasing personal data from an immutable log
Before you start
The concept
Most services store current state: the orders row says status = 'CANCELLED', and how it got there is gone, overwritten by each UPDATE. Event sourcing stores the changes themselves. Every state change is recorded as an immutable event, appended in order: OrderPlaced, PaymentAuthorized, OrderCancelled. The events are the source of truth. Current state is not stored as the primary record. It is derived by starting from nothing and applying the events one after another.
- Aggregate: the unit whose events form one ordered stream and whose rules are checked together. Here, one order.
- Command: a request to change the aggregate (
cancel order). It may be refused. - Event: a fact that a change happened (
OrderCancelled). It is in the past tense and is never refused, edited or deleted. - Replay: load the aggregate's events and fold them into its current state.
- Snapshot: a saved copy of the state at some version, so replay can start there instead of at the first event.
- Projection: a consumer that folds events into a table shaped for queries (a read model). It can be deleted and rebuilt from the events at any time.
Handling a command is always the same three steps: load the events for that order and replay them, decide whether the command is allowed in that state and which event it produces, append the event, on the condition that nobody else appended to this order in the meantime.
Advantages
- A complete history for free: what happened, in which order, and why the state is what it is
- New read models can be built later from old events, for questions nobody had at the time
- State at any past moment can be reconstructed by replaying up to that point
- A bug in derived state is fixed by correcting the code and replaying
- Saving state and producing its event are one append: no dual write inside the service
Disadvantages
- Every query other than 'load one aggregate' needs a projection, which is eventually consistent
- Events live forever, so their schemas must be supported forever
- Erasing personal data from an immutable log needs to be designed in from the start
- Unfamiliar to most developers: more concepts, more code, harder debugging by SQL
- Replays must not repeat side effects such as emails or payments
- Long-lived aggregates need snapshots to stay fast to load
Why it exists
An orders table that is updated in place answers one question: what is the state now? Three kinds of need go unmet.
- Explanation. A customer disputes a cancellation. The row says
CANCELLED. Was the payment declined, did stock run out, did a timeout fire, did support cancel it? Teams add audit tables and log lines to answer this, and those are written by separate code that can be forgotten or drift from the real change. - New questions about the past. The business wants to know how often orders were paid and then cancelled within ten minutes, for the last year. Unless someone stored that transition, the information no longer exists.
- Agreement between state and events. Other services need to hear about every change. With stored state, the change and the event are two writes that must be kept atomic with an outbox. If the event is the stored change, there is only one write.
Event sourcing turns the audit trail into the primary record, so it cannot drift: there is no other record for it to drift from. Martin Fowler's description of the pattern makes the same point about what persists: an application ends up with two things, the application state and the event log, and the state can always be rebuilt from the log. He is also clear that it is an unusual style that many find awkward, to be adopted where it gives a return and not by default.
Kafka looks like the obvious place to keep these events: it is a durable, ordered, replayable log, and other services already read from it. That is half right, and the half that is wrong causes real data bugs. The next section is specific about both halves.
How it works inside
Decide, then append with an expected version
Two requests arrive for order o-1001 at the same moment: support cancels it, and the saga confirms it. Each loads the events, sees a PAID order at version 2, decides its command is valid, and appends. If both appends succeed, the stream says the order was cancelled and then confirmed, which no rule allows. The event store must refuse the second append. The standard mechanism is optimistic concurrency: every append states the version it expects the stream to be at, and the store rejects it if the stream has moved on. The loser reloads, sees the new state and decides again, which this time means refusing the command.
Kafka as the event log: what fits and what does not
| Need | What Kafka gives | Verdict |
|---|---|---|
| Durable, immutable, ordered events | An append-only replicated log. With the order id as key, all events of one order are in one partition in append order | Fits |
| Replay for projections and new consumers | Any consumer group can read a topic from the earliest offset, as long as the records are retained | Fits, if retention is set to keep them (the default is 7 days) |
| Distribute events to other services | Consumer groups, each with its own position | Fits: this is what Kafka is built for |
| Append only if the stream is at version N | A produce request appends. It has no condition on the current end of the log or on the last record for a key | Missing. Two writers with stale state both succeed |
| Load the events of one aggregate | Reads are by partition and offset (or timestamp). There is no lookup by key: finding one order's events means scanning its partition | Missing. Too slow to do per command |
| Read your own write immediately | A projection sees the event after it has been consumed, some time later | Eventual only |
What teams do instead
- An event store in a database, Kafka for distribution. Events are appended to a table with a unique constraint on
(aggregate_id, version), or to a purpose-built event store. The constraint is the concurrency check, and an index gives load-by-aggregate. A relay or CDC connector then publishes the same events to Kafka for projections and other services. This is the design implemented below. The source of truth is the database table, and the Kafka topic is a faithful, replayable copy. - A single writer per aggregate, with Kafka as the log. Commands are sent to a topic keyed by aggregate id. Within a consumer group, one consumer owns each partition, so it is the only process deciding for those aggregates and handles their commands one at a time. It keeps their current state in a local store built from the events, and appends events with a read-process-write transaction. Concurrency control comes from partition ownership instead of a version check. This works, at the price of asynchronous commands (the caller must wait for a result event), state stores to manage, and careful handling of rebalances.
- Not event-sourcing at all. Keep the state table, add an outbox, and publish events. For many services the history that the events topic provides is enough.
Snapshots and compacted topics
Replaying 12 events for an order is instant. An aggregate with 50,000 events, such as a long-lived account, is not. A snapshot stores the state at version N, and loading becomes: read the snapshot, then replay only events after N. A snapshot is a cache. It can be deleted without loss and must be discarded when the replay logic changes.
A compacted topic is the Kafka form of the same idea for consumers: a topic such as orders.state, keyed by order id, with cleanup.policy=compact, where each record is the order's full current state. Compaction retains at least the latest record per key, so a new consumer can load current state without reading all history. Two cautions. Compaction does not promise exactly one record per key: older values remain until the cleaner has processed them, so readers must take the last one. And the event topic must never be compacted: its records share the order id as key, so compaction would delete every event except the most recent one per order.
Projections
A projection is an ordinary Kafka consumer with its own group. It applies each event to a query table and must tolerate redelivery, so each row stores the version of the last event applied and ignores anything not newer. To rebuild it, for a bug fix or a new column, stop it, truncate its table, reset its group's offsets to the earliest position and let it run. That only works while the topic still holds the events, or can be refilled from the event store.
Schema evolution of events
A row in a state table can be migrated. An event written three years ago cannot be rewritten without giving up the claim that the log is the record. So the code that replays must read every version ever written. In practice: only add optional fields with defaults, never change the meaning of an existing field or event type, introduce a new event type when the meaning changes, and convert old shapes to the current one at read time (upcasting). A schema registry with a compatibility rule enforces the first part mechanically (see Schema Registry and schema evolution).
Erasure
A request to erase a person's data conflicts directly with an append-only log. Kafka cannot delete one record from the middle of a non-compacted topic. Options that are used in practice, to be chosen with your legal and security colleagues:
- Keep personal data out of events. Events carry a customer id. Name, address and email live in a normal, mutable table that can be deleted. This is the simplest and should be the default.
- Crypto-shredding. Encrypt personal fields with a key per person, held in a separate key store. Erasure is deleting the key, which leaves the ciphertext in every copy and backup unreadable.
- Limited retention. Give the Kafka topic a finite retention and keep long-term history only in the event store, where a controlled rewrite is at least possible.
- Tombstones on compacted topics. For latest-state topics, a record with the key and a null value removes earlier values once the cleaner runs. The tombstone itself is removed later, after
delete.retention.ms.
Failure scenarios
| Failure point | Resulting state | Recovery |
|---|---|---|
| Service crashes after deciding, before the append commits | No event. The order is unchanged | The caller gets an error and retries the command. State was only in memory. |
| Two commands for one order append the same version | One insert succeeds, the other violates the primary key and rolls back | The loser reloads, replays the new event and decides again. No lost update. |
| Crash after the append commits, before publication to Kafka | Event is in the store, not yet on the topic | The relay or CDC connector publishes it. Projections are behind for a moment, never wrong. |
| Relay publishes an event twice | Two records with the same version on the topic | The projection's version guard ignores the second. |
| Projection crashes after its database commit, before the offset commit | Row updated, offset not advanced | The record is redelivered and ignored: its version is not newer than the row's. |
| A bug in the projection wrote wrong rows | Read model is wrong. Events are untouched | Fix the code, truncate the table, reset the group's offsets to earliest and replay. |
| Snapshot written by old code no longer matches the replay logic | Loading from it gives wrong state | Delete the snapshots. Loading falls back to a full replay, and new snapshots are written. |
| Topic retention removed old events | A new projection sees only part of the history | Refill from the event store, which is the source of truth. If Kafka was the only copy, the history is gone. |
| An old event cannot be deserialised by new code | Loading that order fails. A projection stops at that record | Add an upcaster for the old shape. Never fix it by editing the stored event. |
Architecture
Drawing diagram…
Flowchart: APIs send commands to the order command handler, which loads a snapshot and later events from the event store in the order database, decides using an in-memory order aggregate, and appends a new event with an expected version. A relay or CDC publishes events to the orders.events topic keyed by order id. A projection consumes it and maintains an order summary read model that the order history page queries, and may publish latest state to a compacted topic. Payment and Inventory also consume the events.
flowchart LR
API["Checkout and support APIs"] -->|"command"| CH["Order command handler"]
subgraph ODB["Order database"]
ES["order_event, the event store"]
SN["order_snapshot"]
end
CH -->|"1 load snapshot and later events"| ES
CH -.-> SN
CH -->|"2 decide"| AGG["Order aggregate in memory"]
CH -->|"3 append with expected version"| ES
ES -->|"relay or CDC"| T["Topic orders.events, key orderId"]
T --> PROJ["Order summary projection"]
T --> PAY["Payment Service"]
T --> INV["Inventory Service"]
PROJ --> RM["order_summary read model"]
UI["Order history page"] -->|"query"| RM
PROJ -.->|"optional latest state"| ST["Compacted topic orders.state"]- The command handler never reads Kafka. Loading an order is an indexed query on the event store, and the append is guarded by the primary key
(aggregate_id, version). - The event table is also the outbox: the row that changes the state is the event to publish, so there is nothing to keep in step.
- Reads for screens and searches go to the read model, not to the event store. The read model lags the write by the publication and consumption delay.
- The compacted
orders.statetopic is optional. It serves consumers that want only the latest state of each order without replaying its events.
Message flow
Drawing diagram…
Sequence diagram: a cancel command and a confirm command for the same order both load version 2. The cancel appends version 3 successfully. The confirm's append of version 3 is rejected, so the handler reloads, sees the order is cancelled and refuses the command. The cancelled event is published to Kafka and applied by the projection, and a duplicate delivery of it is ignored because its version is not newer.
sequenceDiagram participant S as Support API participant G as Saga listener participant H as Order command handler participant E as Event store participant K as Kafka orders.events participant P as Summary projection S->>H: cancel o-1001 G->>H: confirm o-1001 H->>E: load events of o-1001, for cancel E-->>H: OrderPlaced v1, PaymentAuthorized v2 H->>E: load events of o-1001, for confirm E-->>H: OrderPlaced v1, PaymentAuthorized v2 H->>E: append OrderCancelled as v3 E-->>H: ok H->>E: append OrderConfirmed as v3 E-->>H: rejected, v3 already exists H->>E: reload events of o-1001 E-->>H: v1, v2, OrderCancelled v3 H-->>G: refused, order is cancelled H-->>S: cancelled E->>K: OrderCancelled v3, key o-1001 K->>P: OrderCancelled v3 P->>P: update order_summary where last_version is below 3 K->>P: OrderCancelled v3 again after a relay retry P->>P: version not newer, ignored
- Both commands were valid against the state they loaded. Only the expected-version check at append time can tell that one of them was working from a stale view.
- The rejected command is not an error to surface as a failure. It is retried against fresh state, and the business rule (a cancelled order cannot be confirmed) gives the real answer.
- The projection does not need an event-id table: the version in each event, compared with the version stored on the row, makes applying it idempotent.
In a real system
In a typical order system that adopts event sourcing, only the Order Service is event-sourced: orders have a real lifecycle, disputes need a history, and the business keeps asking new questions about past orders. Payment and Inventory stay as ordinary state-plus-outbox services. Events are appended to order_event in PostgreSQL and published to orders.events: 12 partitions, replication factor 3, key = order id, cleanup.policy=delete with retention.ms=-1 so projections can always be rebuilt from Kafka. The projection runs as consumer group order-summary-projection and maintains order_summary, which the order history page and support tools query. A second group, order-analytics, builds a different read model from the same events.
| Data | Where it lives | Can it be rebuilt? |
|---|---|---|
| Order events | order_event table (source of truth) | No. This is the record |
| Order events for consumers | Topic orders.events | Yes, from the event table |
| Current state of one order | Computed on load. Optionally order_snapshot | Yes, by replay |
| Order list and search | order_summary read model | Yes, by replaying the topic |
| Latest state per order for other services | Compacted topic orders.state | Yes, by replaying the topic |
| Customer name, address, email | A separate mutable customer table, referenced by id | Not derived. Deleted on an erasure request |
When to use it
- The history is itself a business requirement: ledgers and account balances, order and claim lifecycles, anything audited or disputed.
- You expect to derive new views from past activity, and cannot predict which.
- The aggregate has clear rules and a natural stream of facts, and its streams stay small or snapshot well.
- The team is willing to own projections, event versioning and replay tooling for the life of the system.
When not to use it
- The data is edited in place with little interest in history: profiles, settings, catalogues. A table and an outbox are simpler and sufficient.
- Most use is ad hoc querying and reporting on current state. Every such query would need a projection.
- Users must see their own write in every view immediately, and the screens cannot be built around eventual consistency.
- As a blanket architecture for every service. Apply it to the few aggregates that benefit.
- With Kafka as the only store and several writers per aggregate: there is no check to stop conflicting appends.
Java and Spring Boot
An event-sourced Order Service on Spring Boot 3: a PostgreSQL event store with an expected-version check, the order aggregate rebuilt by replay (Java 21 sealed interface and records), a command service that retries on conflict, and a projection that consumes orders.events. The event rows are published to Kafka by the relay or CDC connector from the first two lessons of this path, with the headers id, eventType and version.
-- Order Service database: the event store. Rows are only ever inserted.CREATE TABLE order_event ( seq bigserial UNIQUE, -- global order, used by the relay id uuid NOT NULL UNIQUE, -- event id aggregatetype varchar(255) NOT NULL DEFAULT 'orders', aggregateid varchar(255) NOT NULL, -- order id, becomes the Kafka key version integer NOT NULL, -- 1, 2, 3 ... per order type varchar(255) NOT NULL, -- 'OrderPlaced', 'OrderCancelled', ... payload jsonb NOT NULL, created_at timestamptz NOT NULL DEFAULT now(), published_at timestamptz, -- used only by a polling relay PRIMARY KEY (aggregateid, version) -- the optimistic concurrency check);-- Load one aggregate (index scan on the primary key):-- SELECT type, payload FROM order_event-- WHERE aggregateid = ? AND version > ? ORDER BY version;-- Append with an expected version. If another transaction already wrote that version,-- the insert violates the primary key and nothing is appended:-- INSERT INTO order_event (id, aggregateid, version, type, payload)-- VALUES (?, ?, <expected version + 1>, ?, ?::jsonb);-- Snapshots: a cache of replayed state. Safe to delete at any time.CREATE TABLE order_snapshot ( aggregateid varchar(255) PRIMARY KEY, version integer NOT NULL, state jsonb NOT NULL);-- Projection database (read side): one row per order, shaped for queries.CREATE TABLE order_summary ( order_id varchar(36) PRIMARY KEY, customer_id varchar(36) NOT NULL, status varchar(16) NOT NULL, total_cents bigint NOT NULL, last_version integer NOT NULL -- version of the last event applied);CREATE INDEX order_summary_by_customer ON order_summary (customer_id);-- Applying an event is guarded by the version, so a redelivered event changes nothing:-- UPDATE order_summary SET status = ?, last_version = ?-- WHERE order_id = ? AND last_version < ?;Why it is written this way
- The primary key
(aggregateid, version)does two jobs: it is the index that makes loading one order fast, and it is the constraint that rejects a second writer. Kafka offers neither, which is why the store is a database. - The table uses the outbox column names from earlier lessons, so it is its own outbox. There is one insert per state change, and it is both the state and the event.
- There is no
UPDATEorDELETEonorder_eventin application code, apart from a polling relay settingpublished_at. With CDC even that goes away. Revoke those privileges from the application's database role if you can. order_summaryhas no foreign keys to the write side and lives wherever queries need it. It can be dropped and rebuilt from the topic.
package com.example.orders.domain;/** Every fact that can happen to an order. Past tense, immutable, never removed from this list. */public sealed interface OrderEvent { String orderId(); record OrderPlaced(String orderId, String customerId, String sku, int quantity, long totalCents) implements OrderEvent {} record PaymentAuthorized(String orderId, long amountCents) implements OrderEvent {} record OrderConfirmed(String orderId) implements OrderEvent {} record OrderCancelled(String orderId, String reason) implements OrderEvent {} /** The name stored in the type column and sent as the eventType header. */ default String typeName() { return getClass().getSimpleName(); } static Class<? extends OrderEvent> classFor(String typeName) { return switch (typeName) { case "OrderPlaced" -> OrderPlaced.class; case "PaymentAuthorized" -> PaymentAuthorized.class; case "OrderConfirmed" -> OrderConfirmed.class; case "OrderCancelled" -> OrderCancelled.class; default -> throw new IllegalArgumentException("Unknown event type " + typeName); }; }}Why it is written this way
- A sealed interface lets the compiler check that every
switchover events handles all of them. Adding an event type produces compile errors in each place that must be updated. - The mapping from stored type name to class is explicit. Deriving it from class names by reflection would turn renaming a class into a change of the stored format.
- Events carry ids, not personal data.
customerIdis a reference to a table that can be changed or deleted. - Evolving an event: add a field as optional and configure the JSON mapper to ignore unknown properties, so old code reads new events and new code reads old ones. A change of meaning is a new event type.
package com.example.orders.domain;import com.example.orders.domain.OrderEvent.OrderCancelled;import com.example.orders.domain.OrderEvent.OrderConfirmed;import com.example.orders.domain.OrderEvent.OrderPlaced;import com.example.orders.domain.OrderEvent.PaymentAuthorized;import java.util.List;/** Current state of one order. Never stored as the record: always the result of applying events. */public record Order(String id, Status status, String customerId, long totalCents, int version) { public enum Status { NEW, PLACED, PAID, CONFIRMED, CANCELLED } public static Order empty(String id) { return new Order(id, Status.NEW, null, 0L, 0); } /** Replay: fold events, oldest first, into state. */ public static Order replay(Order start, List<OrderEvent> events) { Order order = start; for (OrderEvent event : events) { order = order.apply(event); } return order; } /** Applies a fact. No validation here: the event already happened. */ public Order apply(OrderEvent event) { return switch (event) { case OrderPlaced e -> new Order(id, Status.PLACED, e.customerId(), e.totalCents(), version + 1); case PaymentAuthorized e -> new Order(id, Status.PAID, customerId, totalCents, version + 1); case OrderConfirmed e -> new Order(id, Status.CONFIRMED, customerId, totalCents, version + 1); case OrderCancelled e -> new Order(id, Status.CANCELLED, customerId, totalCents, version + 1); }; } /* Decisions: check the rules against current state and return the event, or refuse. */ public OrderEvent place(String customerId, String sku, int quantity, long totalCents) { require(status == Status.NEW, "order already exists"); require(quantity > 0 && totalCents > 0, "quantity and total must be positive"); return new OrderPlaced(id, customerId, sku, quantity, totalCents); } public OrderEvent authorizePayment(long amountCents) { require(status == Status.PLACED, "only a placed order can be paid"); require(amountCents == totalCents, "amount does not match the order total"); return new PaymentAuthorized(id, amountCents); } public OrderEvent confirm() { require(status == Status.PAID, "only a paid order can be confirmed"); return new OrderConfirmed(id); } public OrderEvent cancel(String reason) { require(status == Status.PLACED || status == Status.PAID, "order can no longer be cancelled"); return new OrderCancelled(id, reason); } private static void require(boolean condition, String message) { if (!condition) { throw new IllegalStateException(message); } }}Why it is written this way
applyand the decision methods are deliberately separate.applymust accept every stored event without question, including ones that today's rules would refuse, because it runs on history. Rules belong only in the decision methods.- The aggregate is a pure value with no database or Kafka access, so the whole behaviour of an order can be unit-tested as: given these events, when this command, then that event or that refusal.
versioncounts the events applied. After replay it is the version of the last stored event, which is what the next append will state as its expectation.applymust stay deterministic: no clocks, random values or lookups. Anything variable belongs in the event's payload, decided once at command time.
package com.example.orders.store;import com.example.orders.domain.Order;import com.example.orders.domain.OrderEvent;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.util.List;import java.util.UUID;import org.springframework.dao.DuplicateKeyException;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.stereotype.Repository;import org.springframework.transaction.annotation.Transactional;@Repositorypublic class OrderEventStore { /** Thrown when another writer appended to the same order first. */ public static class ConcurrentAppendException extends RuntimeException { public ConcurrentAppendException(String orderId, int version, Throwable cause) { super("Order " + orderId + " already has version " + version, cause); } } private static final int SNAPSHOT_EVERY = 100; private final JdbcTemplate jdbc; private final ObjectMapper json; public OrderEventStore(JdbcTemplate jdbc, ObjectMapper json) { this.jdbc = jdbc; this.json = json; } /** Rebuilds the order: latest snapshot if there is one, then every later event in order. */ @Transactional(readOnly = true) public Order load(String orderId) { Order start = jdbc.query( "SELECT state::text AS state FROM order_snapshot WHERE aggregateid = ?", (rs, n) -> fromJson(rs.getString("state"), Order.class), orderId) .stream().findFirst().orElse(Order.empty(orderId)); List<OrderEvent> later = jdbc.query(""" SELECT type, payload::text AS payload FROM order_event WHERE aggregateid = ? AND version > ? ORDER BY version""", (rs, n) -> (OrderEvent) fromJson(rs.getString("payload"), OrderEvent.classFor(rs.getString("type"))), orderId, start.version()); return Order.replay(start, later); } /** Appends one event as version expectedVersion + 1, or fails if that version already exists. */ @Transactional public Order append(Order current, OrderEvent event) { int newVersion = current.version() + 1; try { jdbc.update(""" INSERT INTO order_event (id, aggregateid, version, type, payload) VALUES (?, ?, ?, ?, ?::jsonb)""", UUID.randomUUID(), current.id(), newVersion, event.typeName(), toJson(event)); } catch (DuplicateKeyException e) { throw new ConcurrentAppendException(current.id(), newVersion, e); } Order updated = current.apply(event); if (newVersion % SNAPSHOT_EVERY == 0) { jdbc.update(""" INSERT INTO order_snapshot (aggregateid, version, state) VALUES (?, ?, ?::jsonb) ON CONFLICT (aggregateid) DO UPDATE SET version = EXCLUDED.version, state = EXCLUDED.state""", updated.id(), updated.version(), toJson(updated)); } return updated; } private String toJson(Object value) { try { return json.writeValueAsString(value); } catch (JsonProcessingException e) { throw new IllegalStateException("Cannot serialise " + value.getClass().getSimpleName(), e); } } private <T> T fromJson(String text, Class<T> type) { try { return json.readValue(text, type); } catch (JsonProcessingException e) { throw new IllegalStateException("Cannot read stored " + type.getSimpleName(), e); } }}Why it is written this way
- The append states its expectation through the value it inserts:
current.version() + 1. If someone else got there first, the primary key rejects the row and Spring's JDBC exception translation raisesDuplicateKeyException. - Nothing is published here. The inserted row is picked up by the relay or CDC connector, so the event reaches Kafka if and only if it was stored.
- The snapshot is written in the same transaction as the event that reaches the threshold, so it can never be ahead of the events. If the snapshot is missing or deleted,
loadreplays from version 0. - A snapshot is serialised state, so a change to the
Orderrecord or toapplycan make old snapshots wrong. Delete them as part of such a release. They rebuild themselves.
package com.example.orders;import com.example.orders.domain.Order;import com.example.orders.domain.OrderEvent;import com.example.orders.store.OrderEventStore;import com.example.orders.store.OrderEventStore.ConcurrentAppendException;import java.util.function.Function;import org.springframework.stereotype.Service;@Servicepublic class OrderCommandService { private static final int MAX_ATTEMPTS = 3; private final OrderEventStore store; public OrderCommandService(OrderEventStore store) { this.store = store; } public Order place(String orderId, String customerId, String sku, int quantity, long totalCents) { return handle(orderId, order -> order.place(customerId, sku, quantity, totalCents)); } public Order authorizePayment(String orderId, long amountCents) { return handle(orderId, order -> order.authorizePayment(amountCents)); } public Order confirm(String orderId) { return handle(orderId, Order::confirm); } public Order cancel(String orderId, String reason) { return handle(orderId, order -> order.cancel(reason)); } /** Load, decide, append. On a concurrent append, start again from fresh state. */ private Order handle(String orderId, Function<Order, OrderEvent> decide) { for (int attempt = 1; ; attempt++) { Order current = store.load(orderId); OrderEvent event = decide.apply(current); // throws IllegalStateException if the rules refuse try { return store.append(current, event); } catch (ConcurrentAppendException e) { if (attempt == MAX_ATTEMPTS) { throw e; } } } }}Why it is written this way
- The retry repeats the whole cycle, including the decision. Retrying only the insert with a higher version would append an event that was decided against state that no longer exists.
handleis not@Transactional: eachloadandappendis its own transaction. A failed insert aborts a PostgreSQL transaction, so the retry must begin a new one.- After a conflict the decision usually changes: the reloaded order is already cancelled, so
confirmthrowsIllegalStateException, which the API maps to a conflict response. - Commands that arrive from Kafka (for example a saga event that leads to
confirm) can be redelivered. The rules make that safe here: a secondconfirmon a confirmed order is refused, and the listener should treat that refusal as already done.
package com.example.orders.projection;import com.example.orders.domain.OrderEvent;import com.example.orders.domain.OrderEvent.OrderCancelled;import com.example.orders.domain.OrderEvent.OrderConfirmed;import com.example.orders.domain.OrderEvent.OrderPlaced;import com.example.orders.domain.OrderEvent.PaymentAuthorized;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.nio.charset.StandardCharsets;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;/** Read model for the order history page. Rebuildable: truncate the table and replay the topic. */@Componentpublic class OrderSummaryProjection { private final JdbcTemplate jdbc; private final ObjectMapper json; public OrderSummaryProjection(JdbcTemplate jdbc, ObjectMapper json) { this.jdbc = jdbc; this.json = json; } @KafkaListener(id = "order-summary", topics = "orders.events", groupId = "order-summary-projection") @Transactional public void on(ConsumerRecord<String, String> record) throws JsonProcessingException { int version = Integer.parseInt(header(record, "version")); OrderEvent event = json.readValue(record.value(), OrderEvent.classFor(header(record, "eventType"))); switch (event) { case OrderPlaced e -> jdbc.update(""" INSERT INTO order_summary (order_id, customer_id, status, total_cents, last_version) VALUES (?, ?, 'PLACED', ?, ?) ON CONFLICT (order_id) DO NOTHING""", e.orderId(), e.customerId(), e.totalCents(), version); case PaymentAuthorized e -> setStatus(e.orderId(), "PAID", version); case OrderConfirmed e -> setStatus(e.orderId(), "CONFIRMED", version); case OrderCancelled e -> setStatus(e.orderId(), "CANCELLED", version); } } private void setStatus(String orderId, String status, int version) { // 0 rows updated means this event was already applied: a redelivery or a relay duplicate. jdbc.update(""" UPDATE order_summary SET status = ?, last_version = ? WHERE order_id = ? AND last_version < ?""", status, version, orderId, version); } private static String header(ConsumerRecord<String, String> record, String name) { Header h = record.headers().lastHeader(name); if (h == null) { throw new IllegalArgumentException("Missing header " + name); } return new String(h.value(), StandardCharsets.UTF_8); }}Why it is written this way
- The projection is idempotent through
last_version, not through an event-id table. Events of one order arrive in version order within their partition, so 'not newer than what I have' means 'already applied'. - It has its own consumer group. Rebuilding it does not disturb the Payment or Inventory groups reading the same topic.
- It contains no side effects, only writes to its own table. That is what makes replaying it from the earliest offset safe.
- Set
spring.kafka.consumer.auto-offset-reset=earliestfor this group. A projection that starts at the latest offset silently misses every order placed before it was first deployed. - The switch over the sealed interface is exhaustive: a new event type will not compile until this projection decides what to do with it.
Configuration
Topic settings decide whether Kafka can serve as a replayable copy of the events and as a latest-state store. Defaults are from the Kafka 4.1 topic configuration reference. The last two entries are application settings.
retention.msDefault 604800000 (7 days)
- What it does
- How long records are kept on a topic with the delete policy before old segments are discarded.
-1means no time limit. - When to change it
- Set
-1on the event topic if projections must be rebuildable from Kafka alone. Otherwise keep a finite value and rebuild from the event store. Decide deliberately: the default silently deletes history after a week.
cleanup.policyDefault delete
- What it does
deletediscards old segments by time or size.compactretains at least the latest record for each key. Both can be combined.- When to change it
- Keep
deleteon the event topic. Usecompactonly on a latest-state topic such asorders.state. Compacting the event topic destroys the history.
min.cleanable.dirty.ratioDefault 0.5
- What it does
- How much of a compacted log must be uncompacted before the cleaner works on it. It bounds the space taken by superseded records.
- When to change it
- Lower it on a latest-state topic if new consumers take too long to load because they read many outdated values. Lower values mean more cleaning work.
delete.retention.msDefault 86400000 (1 day)
- What it does
- How long tombstones are kept on a compacted topic before they are removed too.
- When to change it
- A consumer reading from the start must finish within this time to be sure of seeing every delete. Raise it if a full read of the topic can take longer than a day.
min.compaction.lag.msDefault 0
- What it does
- Minimum time a record stays uncompacted, which guarantees consumers that long to see each intermediate value.
- When to change it
- Raise it if consumers of a latest-state topic need to observe every update and can lag by a known amount.
auto.offset.resetDefault latest
- What it does
- Where a consumer group with no committed offset starts.
- When to change it
- Use
earliestfor every projection. A read model must be built from the first event, not from whenever it was first started.
snapshot intervalDefault none (application setting, every 100 events in this lesson)
- What it does
- How many events may accumulate before a snapshot of the aggregate is stored.
- When to change it
- Measure load time first. Orders with a dozen events need no snapshots at all. Add them for aggregates whose streams grow without bound.
append retry attemptsDefault none (application setting, 3 in this lesson)
- What it does
- How many times a command is retried from the load step after a concurrent append.
- When to change it
- A few is enough. Frequent conflicts on one aggregate mean it is too large or too contended, and should be split or handled by a single writer.
Hands-on exercise
Replay an order, then watch compaction destroy a history
Rebuild an order's state by hand from its events, then use the compaction simulation to see why an event topic must not be compacted and what a latest-state topic is for.
Steps · 0 of 5 done
Compaction on a topic keyed by order id
Simulation · not a real clusterWrite several records for the same order id, run compaction, and see what is left of that order's history.
What you should see
The replay gives PLACED v1, PAID v2, CANCELLED v3. Both commands are refused: a cancelled order cannot be confirmed or cancelled again. After compaction the cleaned part of the log keeps only the latest record per order id. If those were events, only OrderCancelled would remain and the order could not be rebuilt: no customer, no total. Compaction is safe only on orders.state, where each record carries the order's complete current state, so the latest record alone is enough.
What can go wrong
Kafka as the only event store with several writers
- Symptom
- An order's stream contains impossible sequences, such as
OrderCancelledfollowed byOrderConfirmed, a few milliseconds apart. - Cause
- Two service instances each read the state, decided, and produced. Kafka appended both: a produce request has no expected-version condition, and idempotence or transactions do not add one.
- Fix
- Append to a store that enforces a unique version per aggregate and publish from it, or route all commands for an aggregate through the single consumer that owns its partition.
Compacting the event topic
- Symptom
- A rebuilt projection has orders with a status but no customer or total, or is missing orders. Older events for each key are gone.
- Cause
cleanup.policy=compactwas set on a topic whose records are individual events sharing the aggregate id as key. Compaction kept the latest record per key and removed the rest.- Fix
- Use the delete policy with the retention you intend on event topics. Publish full-state records to a separate compacted topic if consumers need latest state.
History lost to default retention
- Symptom
- A new projection deployed months later only contains orders from the last week.
- Cause
- The event topic was created with the default
retention.msof 7 days, and Kafka was treated as the permanent record. - Fix
- Set retention explicitly when the topic is created. Keep the source of truth in the event store and be able to republish from it.
Side effects during replay
- Symptom
- After a projection rebuild, customers receive confirmation emails for orders from last year, or a downstream system is called thousands of times.
- Cause
- The consumer that was replayed also performs external actions for each event.
- Fix
- Keep projections free of side effects. Run notifications and external calls in separate consumer groups that are never reset, or behind a gateway that is disabled during a replay.
Breaking old events with a schema change
- Symptom
- After a release, loading some older orders fails with a deserialisation error, or totals are wrong for orders placed before a certain date.
- Cause
- A field was renamed, removed or given a new meaning. Stored events still have the old shape, and replay now misreads them.
- Fix
- Additive changes only, with defaults. A new event type for a new meaning. Upcast old versions on read. Test replay against a sample of real historical events before every release.
Reading the projection right after a write
- Symptom
- A user cancels an order, the page reloads, and the order still shows as paid. A second click returns an error.
- Cause
- The page reads
order_summary, which is updated only after the event has been published and consumed. - Fix
- Return the new state from the command response and show it. For checks that must be current, load the aggregate from the event store instead of the read model.
Personal data written into events
- Symptom
- An erasure request arrives, and the customer's name and address are in thousands of immutable records, on the topic, in the store and in backups.
- Cause
- Events were designed as copies of the request, including personal fields.
- Fix
- Store references in events and keep personal data in a deletable table, or encrypt personal fields with a key per person and delete the key on erasure.
How to troubleshoot
An order is in a state nobody can explain
Check
SELECT version, type, payload, created_at FROM order_event WHERE aggregateid = 'o-1001' ORDER BY version;
- How to read it
- The stream is the explanation: each row is one decision with its time. Gaps in
versioncannot occur with the primary key. If the state shown to users differs from what the stream implies, the projection is behind or wrong, not the order.
The read model is behind the event store
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group order-summary-projection
- How to read it
- Lag on
orders.eventsmeans the projection is slow or stopped. No lag while rows are still stale means the events have not reached Kafka: look for unpublished rows in the event table or a failed connector.
A projection must be rebuilt
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --group order-summary-projection --topic orders.events --reset-offsets --to-earliest --execute
- How to read it
- Stop the projection first: offsets can only be reset while the group has no active members. Truncate the read model, run the command, then start the projection and watch its lag fall to zero.
Unsure whether the topic still holds the full history
Check
kafka-configs.sh --bootstrap-server broker:9092 --describe --entity-type topics --entity-name orders.events
- How to read it
- Look at
cleanup.policyandretention.msamong the overrides. No override means the broker defaults apply, which is the delete policy with 7 days unless the cluster changed them.
Many commands fail with ConcurrentAppendException
Check
The application's count of append conflicts per aggregate id (log or metric)
- How to read it
- Occasional conflicts are the mechanism working. A steady stream on the same ids means several actors keep writing to one aggregate: check for a redelivery loop, or redesign so that aggregate has one writer.
Key takeaways
- In event sourcing the events are the record and the current state is computed by replaying them.
- Every command is load, decide, append, and the append must fail if the stream moved since the load.
- Kafka is a durable ordered log per key and an excellent way to distribute and replay events. It has no conditional append and no lookup by aggregate id.
- The common design is an event store with a unique version per aggregate, publishing to Kafka. The alternative is a single writer per aggregate through partition ownership.
- Snapshots and compacted latest-state topics are caches of the history. Never compact the event topic itself.
- Projections must be idempotent, free of side effects and rebuildable, and they are eventually consistent.
- Events are permanent: evolve schemas additively and keep personal data out of them.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
An order service runs four instances and uses the topic orders.events as its only event store. Each instance keeps order state in memory from consuming the topic, validates a command against it and produces the event with acks=all and idempotence enabled. Audit finds 31 orders whose streams contain OrderCancelled directly followed by OrderConfirmed.
What is the correct fix?
Scenario 2 of 3
To save disk, an engineer changes orders.events (key = order id, one record per event) to cleanup.policy=compact. Two weeks later the analytics team deploys a new projection that reads from the earliest offset. It reports 400,000 orders, almost all with status CONFIRMED or CANCELLED and a total of 0.
What happened?
Scenario 3 of 3
A fix to the order summary projection needs a rebuild of 3 years of events. The same Spring application also contains the listener that emails customers on OrderConfirmed, and both listeners share the consumer group order-service-readers. The plan is to reset that group's offsets to earliest tonight.
What must change before running the plan?
References
References: Apache Kafka 4.1 documentation: Design, log compaction ↗ · Apache Kafka 4.1: topic configuration reference (cleanup.policy, retention.ms, delete.retention.ms) ↗ · Apache Kafka 4.1: producer configuration reference (enable.idempotence, transactional.id) ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener annotation ↗ · Martin Fowler: Event Sourcing ↗ · Martin Fowler: What do you mean by 'Event-Driven'? ↗ · Chris Richardson, microservices.io: Event sourcing pattern ↗ · Report a problem