Skip to content

Path 7 · lesson 4 of 8

Advanced35 minApache Kafka 4.x (topic configuration and design docs read for 4.1) · Spring for Apache Kafka 3.3 on Spring Boot 3.x · Java 21 · PostgreSQL as the event store

Event sourcing

Store every change to an order as an event and derive the current state by replaying them. What Kafka does well as the log, where it falls short as the event store, and what teams do instead.

What you will learn

  • Model an order as a sequence of events and rebuild its current state by replaying them
  • Append events with an expected version so two concurrent commands cannot both succeed on stale state
  • Explain what Kafka provides for event sourcing and the two things it lacks, and choose a design that covers them
  • Build a projection from the event stream that is idempotent and can be rebuilt from scratch
  • Use snapshots and compacted topics correctly, without destroying the history they summarise
  • Plan for event schema evolution and for erasing personal data from an immutable log

Before you start

The concept

Most services store current state: the orders row says status = 'CANCELLED', and how it got there is gone, overwritten by each UPDATE. Event sourcing stores the changes themselves. Every state change is recorded as an immutable event, appended in order: OrderPlaced, PaymentAuthorized, OrderCancelled. The events are the source of truth. Current state is not stored as the primary record. It is derived by starting from nothing and applying the events one after another.

  • Aggregate: the unit whose events form one ordered stream and whose rules are checked together. Here, one order.
  • Command: a request to change the aggregate (cancel order). It may be refused.
  • Event: a fact that a change happened (OrderCancelled). It is in the past tense and is never refused, edited or deleted.
  • Replay: load the aggregate's events and fold them into its current state.
  • Snapshot: a saved copy of the state at some version, so replay can start there instead of at the first event.
  • Projection: a consumer that folds events into a table shaped for queries (a read model). It can be deleted and rebuilt from the events at any time.

Handling a command is always the same three steps: load the events for that order and replay them, decide whether the command is allowed in that state and which event it produces, append the event, on the condition that nobody else appended to this order in the meantime.

Advantages

  • A complete history for free: what happened, in which order, and why the state is what it is
  • New read models can be built later from old events, for questions nobody had at the time
  • State at any past moment can be reconstructed by replaying up to that point
  • A bug in derived state is fixed by correcting the code and replaying
  • Saving state and producing its event are one append: no dual write inside the service

Disadvantages

  • Every query other than 'load one aggregate' needs a projection, which is eventually consistent
  • Events live forever, so their schemas must be supported forever
  • Erasing personal data from an immutable log needs to be designed in from the start
  • Unfamiliar to most developers: more concepts, more code, harder debugging by SQL
  • Replays must not repeat side effects such as emails or payments
  • Long-lived aggregates need snapshots to stay fast to load

Why it exists

An orders table that is updated in place answers one question: what is the state now? Three kinds of need go unmet.

  • Explanation. A customer disputes a cancellation. The row says CANCELLED. Was the payment declined, did stock run out, did a timeout fire, did support cancel it? Teams add audit tables and log lines to answer this, and those are written by separate code that can be forgotten or drift from the real change.
  • New questions about the past. The business wants to know how often orders were paid and then cancelled within ten minutes, for the last year. Unless someone stored that transition, the information no longer exists.
  • Agreement between state and events. Other services need to hear about every change. With stored state, the change and the event are two writes that must be kept atomic with an outbox. If the event is the stored change, there is only one write.

Event sourcing turns the audit trail into the primary record, so it cannot drift: there is no other record for it to drift from. Martin Fowler's description of the pattern makes the same point about what persists: an application ends up with two things, the application state and the event log, and the state can always be rebuilt from the log. He is also clear that it is an unusual style that many find awkward, to be adopted where it gives a return and not by default.

Kafka looks like the obvious place to keep these events: it is a durable, ordered, replayable log, and other services already read from it. That is half right, and the half that is wrong causes real data bugs. The next section is specific about both halves.

How it works inside

Decide, then append with an expected version

Two requests arrive for order o-1001 at the same moment: support cancels it, and the saga confirms it. Each loads the events, sees a PAID order at version 2, decides its command is valid, and appends. If both appends succeed, the stream says the order was cancelled and then confirmed, which no rule allows. The event store must refuse the second append. The standard mechanism is optimistic concurrency: every append states the version it expects the stream to be at, and the store rejects it if the stream has moved on. The loser reloads, sees the new state and decides again, which this time means refusing the command.

Kafka as the event log: what fits and what does not

NeedWhat Kafka givesVerdict
Durable, immutable, ordered eventsAn append-only replicated log. With the order id as key, all events of one order are in one partition in append orderFits
Replay for projections and new consumersAny consumer group can read a topic from the earliest offset, as long as the records are retainedFits, if retention is set to keep them (the default is 7 days)
Distribute events to other servicesConsumer groups, each with its own positionFits: this is what Kafka is built for
Append only if the stream is at version NA produce request appends. It has no condition on the current end of the log or on the last record for a keyMissing. Two writers with stale state both succeed
Load the events of one aggregateReads are by partition and offset (or timestamp). There is no lookup by key: finding one order's events means scanning its partitionMissing. Too slow to do per command
Read your own write immediatelyA projection sees the event after it has been consumed, some time laterEventual only

What teams do instead

  • An event store in a database, Kafka for distribution. Events are appended to a table with a unique constraint on (aggregate_id, version), or to a purpose-built event store. The constraint is the concurrency check, and an index gives load-by-aggregate. A relay or CDC connector then publishes the same events to Kafka for projections and other services. This is the design implemented below. The source of truth is the database table, and the Kafka topic is a faithful, replayable copy.
  • A single writer per aggregate, with Kafka as the log. Commands are sent to a topic keyed by aggregate id. Within a consumer group, one consumer owns each partition, so it is the only process deciding for those aggregates and handles their commands one at a time. It keeps their current state in a local store built from the events, and appends events with a read-process-write transaction. Concurrency control comes from partition ownership instead of a version check. This works, at the price of asynchronous commands (the caller must wait for a result event), state stores to manage, and careful handling of rebalances.
  • Not event-sourcing at all. Keep the state table, add an outbox, and publish events. For many services the history that the events topic provides is enough.

Snapshots and compacted topics

Replaying 12 events for an order is instant. An aggregate with 50,000 events, such as a long-lived account, is not. A snapshot stores the state at version N, and loading becomes: read the snapshot, then replay only events after N. A snapshot is a cache. It can be deleted without loss and must be discarded when the replay logic changes.

A compacted topic is the Kafka form of the same idea for consumers: a topic such as orders.state, keyed by order id, with cleanup.policy=compact, where each record is the order's full current state. Compaction retains at least the latest record per key, so a new consumer can load current state without reading all history. Two cautions. Compaction does not promise exactly one record per key: older values remain until the cleaner has processed them, so readers must take the last one. And the event topic must never be compacted: its records share the order id as key, so compaction would delete every event except the most recent one per order.

Projections

A projection is an ordinary Kafka consumer with its own group. It applies each event to a query table and must tolerate redelivery, so each row stores the version of the last event applied and ignores anything not newer. To rebuild it, for a bug fix or a new column, stop it, truncate its table, reset its group's offsets to the earliest position and let it run. That only works while the topic still holds the events, or can be refilled from the event store.

Schema evolution of events

A row in a state table can be migrated. An event written three years ago cannot be rewritten without giving up the claim that the log is the record. So the code that replays must read every version ever written. In practice: only add optional fields with defaults, never change the meaning of an existing field or event type, introduce a new event type when the meaning changes, and convert old shapes to the current one at read time (upcasting). A schema registry with a compatibility rule enforces the first part mechanically (see Schema Registry and schema evolution).

Erasure

A request to erase a person's data conflicts directly with an append-only log. Kafka cannot delete one record from the middle of a non-compacted topic. Options that are used in practice, to be chosen with your legal and security colleagues:

  • Keep personal data out of events. Events carry a customer id. Name, address and email live in a normal, mutable table that can be deleted. This is the simplest and should be the default.
  • Crypto-shredding. Encrypt personal fields with a key per person, held in a separate key store. Erasure is deleting the key, which leaves the ciphertext in every copy and backup unreadable.
  • Limited retention. Give the Kafka topic a finite retention and keep long-term history only in the event store, where a controlled rewrite is at least possible.
  • Tombstones on compacted topics. For latest-state topics, a record with the key and a null value removes earlier values once the cleaner runs. The tombstone itself is removed later, after delete.retention.ms.

Failure scenarios

Failure pointResulting stateRecovery
Service crashes after deciding, before the append commitsNo event. The order is unchangedThe caller gets an error and retries the command. State was only in memory.
Two commands for one order append the same versionOne insert succeeds, the other violates the primary key and rolls backThe loser reloads, replays the new event and decides again. No lost update.
Crash after the append commits, before publication to KafkaEvent is in the store, not yet on the topicThe relay or CDC connector publishes it. Projections are behind for a moment, never wrong.
Relay publishes an event twiceTwo records with the same version on the topicThe projection's version guard ignores the second.
Projection crashes after its database commit, before the offset commitRow updated, offset not advancedThe record is redelivered and ignored: its version is not newer than the row's.
A bug in the projection wrote wrong rowsRead model is wrong. Events are untouchedFix the code, truncate the table, reset the group's offsets to earliest and replay.
Snapshot written by old code no longer matches the replay logicLoading from it gives wrong stateDelete the snapshots. Loading falls back to a full replay, and new snapshots are written.
Topic retention removed old eventsA new projection sees only part of the historyRefill from the event store, which is the source of truth. If Kafka was the only copy, the history is gone.
An old event cannot be deserialised by new codeLoading that order fails. A projection stops at that recordAdd an upcaster for the old shape. Never fix it by editing the stored event.
Crash or conflict at each point, the state it leaves, and how the system recovers.

Architecture

Drawing diagram…

Flowchart: APIs send commands to the order command handler, which loads a snapshot and later events from the event store in the order database, decides using an in-memory order aggregate, and appends a new event with an expected version. A relay or CDC publishes events to the orders.events topic keyed by order id. A projection consumes it and maintains an order summary read model that the order history page queries, and may publish latest state to a compacted topic. Payment and Inventory also consume the events.

flowchart LR
  API["Checkout and support APIs"] -->|"command"| CH["Order command handler"]
  subgraph ODB["Order database"]
    ES["order_event, the event store"]
    SN["order_snapshot"]
  end
  CH -->|"1 load snapshot and later events"| ES
  CH -.-> SN
  CH -->|"2 decide"| AGG["Order aggregate in memory"]
  CH -->|"3 append with expected version"| ES
  ES -->|"relay or CDC"| T["Topic orders.events, key orderId"]
  T --> PROJ["Order summary projection"]
  T --> PAY["Payment Service"]
  T --> INV["Inventory Service"]
  PROJ --> RM["order_summary read model"]
  UI["Order history page"] -->|"query"| RM
  PROJ -.->|"optional latest state"| ST["Compacted topic orders.state"]
Writes go to the event store, which enforces one version per order. Kafka carries the same events to projections and other services.
  • The command handler never reads Kafka. Loading an order is an indexed query on the event store, and the append is guarded by the primary key (aggregate_id, version).
  • The event table is also the outbox: the row that changes the state is the event to publish, so there is nothing to keep in step.
  • Reads for screens and searches go to the read model, not to the event store. The read model lags the write by the publication and consumption delay.
  • The compacted orders.state topic is optional. It serves consumers that want only the latest state of each order without replaying its events.

Message flow

Drawing diagram…

Sequence diagram: a cancel command and a confirm command for the same order both load version 2. The cancel appends version 3 successfully. The confirm's append of version 3 is rejected, so the handler reloads, sees the order is cancelled and refuses the command. The cancelled event is published to Kafka and applied by the projection, and a duplicate delivery of it is ignored because its version is not newer.

sequenceDiagram
  participant S as Support API
  participant G as Saga listener
  participant H as Order command handler
  participant E as Event store
  participant K as Kafka orders.events
  participant P as Summary projection
  S->>H: cancel o-1001
  G->>H: confirm o-1001
  H->>E: load events of o-1001, for cancel
  E-->>H: OrderPlaced v1, PaymentAuthorized v2
  H->>E: load events of o-1001, for confirm
  E-->>H: OrderPlaced v1, PaymentAuthorized v2
  H->>E: append OrderCancelled as v3
  E-->>H: ok
  H->>E: append OrderConfirmed as v3
  E-->>H: rejected, v3 already exists
  H->>E: reload events of o-1001
  E-->>H: v1, v2, OrderCancelled v3
  H-->>G: refused, order is cancelled
  H-->>S: cancelled
  E->>K: OrderCancelled v3, key o-1001
  K->>P: OrderCancelled v3
  P->>P: update order_summary where last_version is below 3
  K->>P: OrderCancelled v3 again after a relay retry
  P->>P: version not newer, ignored
Two commands decide on the same version. The event store accepts one append, and the other reloads and is refused.
  1. Both commands were valid against the state they loaded. Only the expected-version check at append time can tell that one of them was working from a stale view.
  2. The rejected command is not an error to surface as a failure. It is retried against fresh state, and the business rule (a cancelled order cannot be confirmed) gives the real answer.
  3. The projection does not need an event-id table: the version in each event, compared with the version stored on the row, makes applying it idempotent.

In a real system

In a typical order system that adopts event sourcing, only the Order Service is event-sourced: orders have a real lifecycle, disputes need a history, and the business keeps asking new questions about past orders. Payment and Inventory stay as ordinary state-plus-outbox services. Events are appended to order_event in PostgreSQL and published to orders.events: 12 partitions, replication factor 3, key = order id, cleanup.policy=delete with retention.ms=-1 so projections can always be rebuilt from Kafka. The projection runs as consumer group order-summary-projection and maintains order_summary, which the order history page and support tools query. A second group, order-analytics, builds a different read model from the same events.

DataWhere it livesCan it be rebuilt?
Order eventsorder_event table (source of truth)No. This is the record
Order events for consumersTopic orders.eventsYes, from the event table
Current state of one orderComputed on load. Optionally order_snapshotYes, by replay
Order list and searchorder_summary read modelYes, by replaying the topic
Latest state per order for other servicesCompacted topic orders.stateYes, by replaying the topic
Customer name, address, emailA separate mutable customer table, referenced by idNot derived. Deleted on an erasure request

When to use it

  • The history is itself a business requirement: ledgers and account balances, order and claim lifecycles, anything audited or disputed.
  • You expect to derive new views from past activity, and cannot predict which.
  • The aggregate has clear rules and a natural stream of facts, and its streams stay small or snapshot well.
  • The team is willing to own projections, event versioning and replay tooling for the life of the system.

When not to use it

  • The data is edited in place with little interest in history: profiles, settings, catalogues. A table and an outbox are simpler and sufficient.
  • Most use is ad hoc querying and reporting on current state. Every such query would need a projection.
  • Users must see their own write in every view immediately, and the screens cannot be built around eventual consistency.
  • As a blanket architecture for every service. Apply it to the few aggregates that benefit.
  • With Kafka as the only store and several writers per aggregate: there is no check to stop conflicting appends.

Java and Spring Boot

An event-sourced Order Service on Spring Boot 3: a PostgreSQL event store with an expected-version check, the order aggregate rebuilt by replay (Java 21 sealed interface and records), a command service that retries on conflict, and a projection that consumes orders.events. The event rows are published to Kafka by the relay or CDC connector from the first two lessons of this path, with the headers id, eventType and version.

src/main/resources/db/migration/V1__event_store.sqlSQL
-- Order Service database: the event store. Rows are only ever inserted.CREATE TABLE order_event (    seq           bigserial    UNIQUE,                -- global order, used by the relay    id            uuid         NOT NULL UNIQUE,       -- event id    aggregatetype varchar(255) NOT NULL DEFAULT 'orders',    aggregateid   varchar(255) NOT NULL,              -- order id, becomes the Kafka key    version       integer      NOT NULL,              -- 1, 2, 3 ... per order    type          varchar(255) NOT NULL,              -- 'OrderPlaced', 'OrderCancelled', ...    payload       jsonb        NOT NULL,    created_at    timestamptz  NOT NULL DEFAULT now(),    published_at  timestamptz,                        -- used only by a polling relay    PRIMARY KEY (aggregateid, version)                -- the optimistic concurrency check);-- Load one aggregate (index scan on the primary key):--   SELECT type, payload FROM order_event--   WHERE aggregateid = ? AND version > ? ORDER BY version;-- Append with an expected version. If another transaction already wrote that version,-- the insert violates the primary key and nothing is appended:--   INSERT INTO order_event (id, aggregateid, version, type, payload)--   VALUES (?, ?, <expected version + 1>, ?, ?::jsonb);-- Snapshots: a cache of replayed state. Safe to delete at any time.CREATE TABLE order_snapshot (    aggregateid varchar(255) PRIMARY KEY,    version     integer      NOT NULL,    state       jsonb        NOT NULL);-- Projection database (read side): one row per order, shaped for queries.CREATE TABLE order_summary (    order_id     varchar(36) PRIMARY KEY,    customer_id  varchar(36) NOT NULL,    status       varchar(16) NOT NULL,    total_cents  bigint      NOT NULL,    last_version integer     NOT NULL                 -- version of the last event applied);CREATE INDEX order_summary_by_customer ON order_summary (customer_id);-- Applying an event is guarded by the version, so a redelivered event changes nothing:--   UPDATE order_summary SET status = ?, last_version = ?--   WHERE order_id = ? AND last_version < ?;

Why it is written this way

  • The primary key (aggregateid, version) does two jobs: it is the index that makes loading one order fast, and it is the constraint that rejects a second writer. Kafka offers neither, which is why the store is a database.
  • The table uses the outbox column names from earlier lessons, so it is its own outbox. There is one insert per state change, and it is both the state and the event.
  • There is no UPDATE or DELETE on order_event in application code, apart from a polling relay setting published_at. With CDC even that goes away. Revoke those privileges from the application's database role if you can.
  • order_summary has no foreign keys to the write side and lives wherever queries need it. It can be dropped and rebuilt from the topic.

Configuration

Topic settings decide whether Kafka can serve as a replayable copy of the events and as a latest-state store. Defaults are from the Kafka 4.1 topic configuration reference. The last two entries are application settings.

retention.msDefault 604800000 (7 days)
What it does
How long records are kept on a topic with the delete policy before old segments are discarded. -1 means no time limit.
When to change it
Set -1 on the event topic if projections must be rebuildable from Kafka alone. Otherwise keep a finite value and rebuild from the event store. Decide deliberately: the default silently deletes history after a week.
cleanup.policyDefault delete
What it does
delete discards old segments by time or size. compact retains at least the latest record for each key. Both can be combined.
When to change it
Keep delete on the event topic. Use compact only on a latest-state topic such as orders.state. Compacting the event topic destroys the history.
min.cleanable.dirty.ratioDefault 0.5
What it does
How much of a compacted log must be uncompacted before the cleaner works on it. It bounds the space taken by superseded records.
When to change it
Lower it on a latest-state topic if new consumers take too long to load because they read many outdated values. Lower values mean more cleaning work.
delete.retention.msDefault 86400000 (1 day)
What it does
How long tombstones are kept on a compacted topic before they are removed too.
When to change it
A consumer reading from the start must finish within this time to be sure of seeing every delete. Raise it if a full read of the topic can take longer than a day.
min.compaction.lag.msDefault 0
What it does
Minimum time a record stays uncompacted, which guarantees consumers that long to see each intermediate value.
When to change it
Raise it if consumers of a latest-state topic need to observe every update and can lag by a known amount.
auto.offset.resetDefault latest
What it does
Where a consumer group with no committed offset starts.
When to change it
Use earliest for every projection. A read model must be built from the first event, not from whenever it was first started.
snapshot intervalDefault none (application setting, every 100 events in this lesson)
What it does
How many events may accumulate before a snapshot of the aggregate is stored.
When to change it
Measure load time first. Orders with a dozen events need no snapshots at all. Add them for aggregates whose streams grow without bound.
append retry attemptsDefault none (application setting, 3 in this lesson)
What it does
How many times a command is retried from the load step after a concurrent append.
When to change it
A few is enough. Frequent conflicts on one aggregate mean it is too large or too contended, and should be split or handled by a single writer.

Hands-on exercise

Replay an order, then watch compaction destroy a history

Rebuild an order's state by hand from its events, then use the compaction simulation to see why an event topic must not be compacted and what a latest-state topic is for.

Steps · 0 of 5 done

Compaction on a topic keyed by order id

Simulation · not a real cluster

Write several records for the same order id, run compaction, and see what is left of that order's history.

What can go wrong

Kafka as the only event store with several writers
Symptom
An order's stream contains impossible sequences, such as OrderCancelled followed by OrderConfirmed, a few milliseconds apart.
Cause
Two service instances each read the state, decided, and produced. Kafka appended both: a produce request has no expected-version condition, and idempotence or transactions do not add one.
Fix
Append to a store that enforces a unique version per aggregate and publish from it, or route all commands for an aggregate through the single consumer that owns its partition.
Compacting the event topic
Symptom
A rebuilt projection has orders with a status but no customer or total, or is missing orders. Older events for each key are gone.
Cause
cleanup.policy=compact was set on a topic whose records are individual events sharing the aggregate id as key. Compaction kept the latest record per key and removed the rest.
Fix
Use the delete policy with the retention you intend on event topics. Publish full-state records to a separate compacted topic if consumers need latest state.
History lost to default retention
Symptom
A new projection deployed months later only contains orders from the last week.
Cause
The event topic was created with the default retention.ms of 7 days, and Kafka was treated as the permanent record.
Fix
Set retention explicitly when the topic is created. Keep the source of truth in the event store and be able to republish from it.
Side effects during replay
Symptom
After a projection rebuild, customers receive confirmation emails for orders from last year, or a downstream system is called thousands of times.
Cause
The consumer that was replayed also performs external actions for each event.
Fix
Keep projections free of side effects. Run notifications and external calls in separate consumer groups that are never reset, or behind a gateway that is disabled during a replay.
Breaking old events with a schema change
Symptom
After a release, loading some older orders fails with a deserialisation error, or totals are wrong for orders placed before a certain date.
Cause
A field was renamed, removed or given a new meaning. Stored events still have the old shape, and replay now misreads them.
Fix
Additive changes only, with defaults. A new event type for a new meaning. Upcast old versions on read. Test replay against a sample of real historical events before every release.
Reading the projection right after a write
Symptom
A user cancels an order, the page reloads, and the order still shows as paid. A second click returns an error.
Cause
The page reads order_summary, which is updated only after the event has been published and consumed.
Fix
Return the new state from the command response and show it. For checks that must be current, load the aggregate from the event store instead of the read model.
Personal data written into events
Symptom
An erasure request arrives, and the customer's name and address are in thousands of immutable records, on the topic, in the store and in backups.
Cause
Events were designed as copies of the request, including personal fields.
Fix
Store references in events and keep personal data in a deletable table, or encrypt personal fields with a key per person and delete the key on erasure.

How to troubleshoot

An order is in a state nobody can explain

Check

SELECT version, type, payload, created_at FROM order_event WHERE aggregateid = 'o-1001' ORDER BY version;
How to read it
The stream is the explanation: each row is one decision with its time. Gaps in version cannot occur with the primary key. If the state shown to users differs from what the stream implies, the projection is behind or wrong, not the order.
The read model is behind the event store

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group order-summary-projection
How to read it
Lag on orders.events means the projection is slow or stopped. No lag while rows are still stale means the events have not reached Kafka: look for unpublished rows in the event table or a failed connector.
A projection must be rebuilt

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --group order-summary-projection --topic orders.events --reset-offsets --to-earliest --execute
How to read it
Stop the projection first: offsets can only be reset while the group has no active members. Truncate the read model, run the command, then start the projection and watch its lag fall to zero.
Unsure whether the topic still holds the full history

Check

kafka-configs.sh --bootstrap-server broker:9092 --describe --entity-type topics --entity-name orders.events
How to read it
Look at cleanup.policy and retention.ms among the overrides. No override means the broker defaults apply, which is the delete policy with 7 days unless the cluster changed them.
Many commands fail with ConcurrentAppendException

Check

The application's count of append conflicts per aggregate id (log or metric)
How to read it
Occasional conflicts are the mechanism working. A steady stream on the same ids means several actors keep writing to one aggregate: check for a redelivery loop, or redesign so that aggregate has one writer.

Key takeaways

  • In event sourcing the events are the record and the current state is computed by replaying them.
  • Every command is load, decide, append, and the append must fail if the stream moved since the load.
  • Kafka is a durable ordered log per key and an excellent way to distribute and replay events. It has no conditional append and no lookup by aggregate id.
  • The common design is an event store with a unique version per aggregate, publishing to Kafka. The alternative is a single writer per aggregate through partition ownership.
  • Snapshots and compacted latest-state topics are caches of the history. Never compact the event topic itself.
  • Projections must be idempotent, free of side effects and rebuildable, and they are eventually consistent.
  • Events are permanent: evolve schemas additively and keep personal data out of them.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

An order service runs four instances and uses the topic orders.events as its only event store. Each instance keeps order state in memory from consuming the topic, validates a command against it and produces the event with acks=all and idempotence enabled. Audit finds 31 orders whose streams contain OrderCancelled directly followed by OrderConfirmed.

What is the correct fix?

Scenario 2 of 3

To save disk, an engineer changes orders.events (key = order id, one record per event) to cleanup.policy=compact. Two weeks later the analytics team deploys a new projection that reads from the earliest offset. It reports 400,000 orders, almost all with status CONFIRMED or CANCELLED and a total of 0.

What happened?

Scenario 3 of 3

A fix to the order summary projection needs a rebuild of 3 years of events. The same Spring application also contains the listener that emails customers on OrderConfirmed, and both listeners share the consumer group order-service-readers. The plan is to reset that group's offsets to earliest tonight.

What must change before running the plan?

References

References: Apache Kafka 4.1 documentation: Design, log compaction ↗ · Apache Kafka 4.1: topic configuration reference (cleanup.policy, retention.ms, delete.retention.ms) ↗ · Apache Kafka 4.1: producer configuration reference (enable.idempotence, transactional.id) ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener annotation ↗ · Martin Fowler: Event Sourcing ↗ · Martin Fowler: What do you mean by 'Event-Driven'? ↗ · Chris Richardson, microservices.io: Event sourcing pattern ↗ · Report a problem