Skip to content

Path 1 · lesson 5 of 8

Intermediate30 minPostgreSQL 18.7 (one primary, two replicas) · Apache Cassandra 5.0.9 (three nodes, replication factor 3), all in Docker

Consistency models and quorums

What a reader may see when data has several copies, why overlapping quorums (R + W > N) give fresh reads, and what each choice costs when replicas are down.

What you will learn

  • Name the consistency guarantee a given read path gives, from eventual to linearizable
  • Use R + W > N to choose read and write consistency levels in Cassandra
  • Explain how a quorum read detects and repairs a stale replica
  • Predict which requests fail when replicas are down, for each choice

Before you start

The concept

A consistency model is a promise about what reads may return when data is copied to several places and changed concurrently. Stronger promises are easier to program against and cost more: more machines must answer each request, so requests are slower and fail more often when machines are down.

ModelWhat a read may returnExample in this lesson
EventualAny value that was written, possibly old; copies converge if writes stopCassandra read at ONE; a Postgres async replica
Read-your-writesAt least your own latest writeReading from the Postgres primary after writing
Fresh per key (overlapping quorums)The latest acknowledged write for that keyCassandra QUORUM writes and QUORUM reads
LinearizableAs if there were one copy and every operation happened at one instantPostgres primary; Cassandra lightweight transactions (SERIAL)
From weakest to strongest. Jepsen's consistency map lists many more, with exact definitions.

Why it exists

Every replicated system answers the same question for every read: how many copies must agree before I answer? Ask one and you are fast and available but may be wrong. Ask all and you are right but unavailable when any copy is down. Quorums are the middle, and choosing them per request is how a Cassandra cluster serves both a shopping cart and an analytics feed.

For a Staff engineer the value is in stating the guarantee precisely in a design review: "reads at ONE after writes at QUORUM can return the previous value for a few hundred milliseconds" is a decision the product owner can accept or reject. "It's eventually consistent" is not.

How it works inside

The quorum rule: R + W > N

With N replicas, a write acknowledged by W of them and a read that asks R of them are guaranteed to overlap when R + W > N. The Cassandra documentation describes its consistency levels as "a version of Dynamo's R + W > N consistency mechanism". With N = 3:

Write level (W)Read level (R)R + W > 3?Read sees the latest acknowledged write?Survives one replica down?
ONE (1)ONE (1)NoNot guaranteedReads and writes
QUORUM (2)ONE (1)NoNot guaranteedReads and writes
QUORUM (2)QUORUM (2)YesYesReads and writes
ALL (3)ONE (1)YesYesReads only; writes fail
ONE (1)ALL (3)YesYesWrites only; reads fail
QUORUM plus QUORUM is the usual choice: fresh reads, and one replica can be down.

A stale read at ONE

The lab's three-node cluster turns hints off (they would otherwise hide the problem, as the previous lesson showed), stops node c3, writes items = 3 at QUORUM to c1 and c2, then brings c3 back. c3 still holds the previous value, 2. A read at ONE coordinated by c3:

Read traceRead at ONE, coordinated by c3, traced

Real database output
Written at QUORUM (c1 and c2) while c3 was down: items = 3
Read at ONE, coordinated by c3:
items
-------
2
(1 rows)
Executing single-partition query on carts
Read 1 live rows and 0 tombstone cells
Request complete
2, the old value, with no error and no warning. The trace shows one local read and no message to any other node: ONE asked a single replica, and it happened to be the stale one. W = 2 plus R = 1 is not more than 3.

A quorum read finds the difference, and repairs it

Read traceRead at QUORUM, coordinated by c3, traced

Real database output
items
-------
3
(1 rows)
reading digest from /c1:7000
Sending READ_REQ message to bb-c1/c1:7000 message size 91 bytes
Read 1 live rows and 0 tombstone cells
Read 1 live rows and 0 tombstone cells
Sending READ_RSP message to /c3:7000 message size 49 bytes
Digest mismatch: Mismatch for key DecoratedKey(5699955792253506986, 616c696365)
c3 read its own copy and asked c1 for a digest, a hash of c1's copy. The hashes differed, so c3 fetched the full data and returned the newest by write timestamp: 3.

Read traceThe repair, later in the same trace

Real database output
Sending read-repair-mutation to Full(/c3:7000,(5357399090202236893,5810918674640069629])
Read 0 live rows and 0 tombstone cells
Sending READ_REPAIR_REQ message to /c3:7000 message size 81 bytes
READ_REPAIR_REQ message received from /c3:7000
Sending READ_REPAIR_RSP message to /c3:7000 message size 32 bytes
READ_REPAIR_RSP message received from /c3:7000
Request complete
Before answering, the coordinator wrote the newest value back to the stale replica: read repair. In Cassandra 5.0 the default table setting read_repair = 'BLOCKING' makes the client wait for it, which the documentation says gives monotonic quorum reads: once a quorum read has returned a value, later quorum reads do not go back to an older one.

Database outputRead at ONE on c3 again, after the quorum read

Real database output
Read at ONE on c3 again, after the QUORUM read:
Consistency level set to ONE.
items
-------
3
c3's copy is now fixed. Read repair fixes only the rows that are read at a level above ONE; rows nobody reads stay stale until a full repair (the Cassandra path covers it).

The price: availability

Stronger reads need more replicas alive. With two of three nodes stopped:

Database outputTwo of three replicas down

Real database output
DN c2 100.0%
UN c1 100.0%
DN c3 100.0%
Read at QUORUM:
code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={'consistency': 'QUORUM', 'required_replicas': 2, 'alive_replicas': 1}
Consistency level set to QUORUM.
Read at ONE:
Consistency level set to ONE.
items
-------
3
QUORUM refuses: it needs 2 and only 1 is alive. ONE still answers. Here it is the right value, but nothing guarantees that; this is the choice the CAP theorem describes, made per request.

Postgres: quorum for durability, one primary for reads

Postgres reads and writes on the primary are linearizable: there is one copy that takes writes. Quorums appear in its synchronous replication. synchronous_standby_names = 'ANY 1 (r1, r2)' makes each commit wait for any one of the two replicas, so a commit is on at least two machines, and one replica can fail without stopping writes.

Database outputQuorum commit with r1 down

Real database output
 application_name | sync_state
------------------+------------
 r2               | quorum
(1 row)

Time: 4.790 ms
r1 is still stopped from the previous lesson's test, where a single synchronous replica made the same kind of insert wait indefinitely. With a quorum of one out of two, r2's acknowledgement is enough: the insert committed in under 5 ms.

Reads from Postgres replicas remain eventually consistent, whatever the commit setting, unless the commit used remote_apply and the read goes to a replica that applied it. Postgres has no per-read quorum: the replica you read from is the one you get.

Architecture

Drawing diagram…

Diagram. A write at quorum reaches c1 and c2, which now hold 3, and misses c3, which holds 2. A read at ONE through c3 asks only c3 and gets 2. A read at quorum through c3 asks c3 and c1; the digests differ, the newer value 3 wins and is written back to c3.

flowchart LR
  W["Write at QUORUM: items = 3"] --> A["c1 has 3"]
  W --> B["c2 has 3"]
  W -.->|missed, node was down| C["c3 has 2"]
  R1["Read at ONE via c3"] --> C
  R2["Read at QUORUM via c3"] --> C
  R2 --> A
  A -. "digest mismatch, then read repair" .-> C
Why ONE can be stale and QUORUM cannot, with N = 3.
  • The write reached two replicas and the quorum read asked two: they must share at least one, here c1.
  • The read at ONE asked only c3, which is in neither the write's set nor any overlap guarantee.
  • Read repair makes the quorum read also heal the replica it found stale, but only for the rows it read.

Step by step

Drawing diagram…

Sequence diagram. The client reads at quorum through c3. c3 reads its local copy, 2, and asks c1 for a digest. The digests differ, so c3 requests the full data from c1, which returns 3 with a newer timestamp. c3 writes 3 to itself as read repair and returns 3 to the client.

sequenceDiagram
  participant App as Client
  participant C3 as c3 (coordinator, stale)
  participant C1 as c1 (replica)
  App->>C3: SELECT at QUORUM
  C3->>C3: read local copy, items = 2
  C3->>C1: digest request
  C1-->>C3: digest of items = 3
  Note over C3: digests differ
  C3->>C1: full data request
  C1-->>C3: items = 3, newer timestamp
  C3->>C3: read repair: write items = 3 locally
  C3-->>App: items = 3
The quorum read from the lab, step by step.
  1. Digests keep the common case cheap: when replicas agree, only one sends the full data.
  2. Conflicts are resolved by write timestamp, per column. The newest timestamp wins, even if it came from a skewed clock.
  3. With read_repair = 'BLOCKING' (the default), the client waits for the repair, which is what makes a later quorum read unable to return the older value.

In a real system

  • Default to QUORUM for both reads and writes (LOCAL_QUORUM in multi-data-centre clusters, so requests do not wait on another region). Weaken a specific path only after stating what it may return.
  • Use ONE where staleness is harmless: view counters, recommendations, logs. Write it down next to the query.
  • Know your replication factor. With RF 3, QUORUM tolerates one replica down; with RF 2, QUORUM is 2 of 2 and tolerates none.
  • In Postgres, use ANY 1 of two or more replicas for durable commits that survive a replica failing, and read your own writes from the primary.
  • Linearizability is per operation and costly. Use it (Postgres primary, Cassandra lightweight transactions) for the few decisions that need it: uniqueness, balances, leases.

Wrong, then right

A shopping-cart service on Cassandra (RF 3) writes cart changes at ONE for speed and reads at ONE. Customers report items disappearing from their cart and reappearing.

Incorrect implementation

CartRepository.javaJava
SimpleStatement add = SimpleStatement    .newInstance("UPDATE carts SET items = items + ? WHERE user_id = ?", added, userId)    .setConsistencyLevel(DefaultConsistencyLevel.ONE);SimpleStatement read = SimpleStatement    .newInstance("SELECT items FROM carts WHERE user_id = ?", userId)    .setConsistencyLevel(DefaultConsistencyLevel.ONE);

Why it is written this way

  • W = 1 and R = 1 with N = 3: 1 + 1 is not more than 3, so a read can miss the latest write.
  • Each request may be coordinated by a different node and served by a different replica.

Why it fails

  • A customer adds an item (acknowledged by one replica), then the cart page reads from another replica that has not received it yet: the item is gone. A refresh hits a third replica: it is back.
  • If the one replica that acknowledged the write fails before the others receive it, the write can be lost entirely.

Corrected implementation

CartRepository.javaJava
// N = 3: QUORUM (2) + QUORUM (2) > 3, so a read always overlaps the latest acknowledged write.SimpleStatement add = SimpleStatement    .newInstance("UPDATE carts SET items = items + ? WHERE user_id = ?", added, userId)    .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM);SimpleStatement read = SimpleStatement    .newInstance("SELECT items FROM carts WHERE user_id = ?", userId)    .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM);

Why it is written this way

  • Every acknowledged add is on two replicas, and every read asks two, so they overlap.
  • LOCAL_QUORUM counts replicas in the coordinator's data centre only, so a cart request never waits on another region.

Trade-offs

  • Each request now waits for the second-fastest replica instead of the fastest, which raises tail latency.
  • With two of three replicas down in the data centre, cart reads and writes fail instead of returning possibly wrong data. For a cart, failing is the right choice.
  • Concurrent adds from two devices are still settled per column by timestamp; a set column (items + ?) merges them, a plain value would not.

Configuration

Where consistency is chosen. Defaults are those of PostgreSQL 18, Cassandra 5.0 and the DataStax Java driver 4.

Consistency level per request (Cassandra driver)Default LOCAL_ONE (driver default)
What it does
How many replicas must answer a read or acknowledge a write.
When to change it
Set it explicitly per statement or per profile; do not rely on the driver default for data that must be fresh.
read_repair (Cassandra table option)Default BLOCKING
What it does
Whether a read above ONE that finds differing replicas repairs them before answering.
When to change it
Leave it; NONE trades monotonic quorum reads for slightly faster reads.
replication (Cassandra keyspace)Default Set per keyspace
What it does
N, the number of replicas per data centre.
When to change it
Use NetworkTopologyStrategy with 3 per data centre for production data, so QUORUM tolerates one replica down.
synchronous_standby_names (Postgres)Default '' (none)
What it does
With ANY n (list), a commit waits for any n of the listed replicas.
When to change it
ANY 1 of two or more replicas makes every commit durable on two machines without a single replica able to stop writes.

Hands-on exercise

Make a replica stale, then catch it

Run the lab and see, on a real three-node Cassandra 5.0 cluster, a stale read at ONE, a quorum read detecting it with a digest mismatch and repairing it, and QUORUM refusing to answer when two of three replicas are down; and Postgres committing with a quorum of replicas while one is down.

Steps · 0 of 5 done

What can go wrong

QUORUM with replication factor 2
Symptom
Every request fails while one node is restarted for an upgrade.
Cause
QUORUM of 2 is 2: it needs both replicas.
Fix
Use replication factor 3 for data read or written at QUORUM.
QUORUM across data centres
Symptom
Latency jumps by a cross-region round trip, and requests fail when the link between regions is slow.
Cause
QUORUM counts replicas in all data centres, so it often has to wait for a remote one.
Fix
Use LOCAL_QUORUM, and accept that another region may briefly see older data.
Assuming a failed write was not applied
Symptom
A write that timed out at QUORUM shows up later.
Cause
It reached fewer replicas than required before the timeout but was not rolled back; repair spreads it.
Fix
Treat timeouts as unknown, not failed; make writes idempotent so a retry is harmless.
Last write wins with skewed clocks
Symptom
An older update overwrites a newer one.
Cause
Conflicts are settled by write timestamp, set by the client or coordinator clock; a clock ahead wins.
Fix
Keep clocks synchronised (NTP), and use lightweight transactions where the order of updates matters.

How to troubleshoot

Clients see Unavailable errors

Check

nodetool status <keyspace>
How to read it
The error's required_replicas and alive_replicas tell you how many replicas were down. Compare with the keyspace's replication factor and the consistency level used.
Users see old values intermittently in Cassandra

Check

Trace a read at the level the application uses (cqlsh: CONSISTENCY ONE; TRACING ON; SELECT ...)
How to read it
A read at ONE that touches one replica can be stale; digest mismatches in quorum traces show replicas are diverging, which points at missed hints or overdue repair.
Postgres commits slow down or hang when one replica has a problem

Check

SHOW synchronous_standby_names; SELECT application_name, sync_state FROM pg_stat_replication;
How to read it
A single name, or FIRST 1 with only one healthy listed replica, makes that replica a dependency of every commit.

Key takeaways

  • Consistency models range from eventual to linearizable; name the one each read path gives.
  • In Cassandra, reads see the latest acknowledged write when R + W > N; QUORUM plus QUORUM with N = 3 is the usual choice.
  • A read at ONE can silently return an old value; a quorum read detects the difference with digests and repairs it.
  • Stronger levels fail sooner when replicas are down: with two of three down, QUORUM is unavailable and ONE still answers.
  • Postgres is linearizable on the primary; ANY n quorum commit makes writes durable on several replicas without one replica stopping commits.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A Cassandra keyspace has replication factor 3. A service writes user settings at ONE and reads them at QUORUM. A user changes a setting and reloads.

Is the reload guaranteed to show the change?

Scenario 2 of 3

A payments ledger on Cassandra uses replication factor 3 in each of two data centres and writes at QUORUM. Write latency doubled after the second data centre was added, and spikes whenever the link between them is slow.

What change keeps fresh local reads and removes the cross-region wait?

Scenario 3 of 3

A Postgres primary has two replicas. The team wants every commit to be on at least two machines, but a single replica failing must not stop writes.

Which setting achieves that?

References

References: Apache Cassandra 5.0 documentation: Dynamo (tunable consistency, R + W > N) ↗ · Apache Cassandra 5.0 documentation: Read repair (BLOCKING and monotonic quorum reads) ↗ · Apache Cassandra 5.0 documentation: Guarantees ↗ · Apache Cassandra 5.0 documentation: Repair ↗ · PostgreSQL 18 documentation: Replication settings (synchronous_standby_names, ANY and FIRST) ↗ · DeCandia et al., Dynamo: Amazon's Highly Available Key-value Store (SOSP 2007) ↗ · Jepsen: Consistency models ↗ · Herlihy and Wing, Linearizability: A Correctness Condition for Concurrent Objects (1990) ↗ · Report a problem