Path 1 · lesson 5 of 8
Consistency models and quorums
What a reader may see when data has several copies, why overlapping quorums (R + W > N) give fresh reads, and what each choice costs when replicas are down.
What you will learn
- Name the consistency guarantee a given read path gives, from eventual to linearizable
- Use R + W > N to choose read and write consistency levels in Cassandra
- Explain how a quorum read detects and repairs a stale replica
- Predict which requests fail when replicas are down, for each choice
Before you start
The concept
A consistency model is a promise about what reads may return when data is copied to several places and changed concurrently. Stronger promises are easier to program against and cost more: more machines must answer each request, so requests are slower and fail more often when machines are down.
| Model | What a read may return | Example in this lesson |
|---|---|---|
| Eventual | Any value that was written, possibly old; copies converge if writes stop | Cassandra read at ONE; a Postgres async replica |
| Read-your-writes | At least your own latest write | Reading from the Postgres primary after writing |
| Fresh per key (overlapping quorums) | The latest acknowledged write for that key | Cassandra QUORUM writes and QUORUM reads |
| Linearizable | As if there were one copy and every operation happened at one instant | Postgres primary; Cassandra lightweight transactions (SERIAL) |
Why it exists
Every replicated system answers the same question for every read: how many copies must agree before I answer? Ask one and you are fast and available but may be wrong. Ask all and you are right but unavailable when any copy is down. Quorums are the middle, and choosing them per request is how a Cassandra cluster serves both a shopping cart and an analytics feed.
For a Staff engineer the value is in stating the guarantee precisely in a design review: "reads at ONE after writes at QUORUM can return the previous value for a few hundred milliseconds" is a decision the product owner can accept or reject. "It's eventually consistent" is not.
How it works inside
The quorum rule: R + W > N
With N replicas, a write acknowledged by W of them and a read that asks R of them are guaranteed to overlap when R + W > N. The Cassandra documentation describes its consistency levels as "a version of Dynamo's R + W > N consistency mechanism". With N = 3:
| Write level (W) | Read level (R) | R + W > 3? | Read sees the latest acknowledged write? | Survives one replica down? |
|---|---|---|---|---|
| ONE (1) | ONE (1) | No | Not guaranteed | Reads and writes |
| QUORUM (2) | ONE (1) | No | Not guaranteed | Reads and writes |
| QUORUM (2) | QUORUM (2) | Yes | Yes | Reads and writes |
| ALL (3) | ONE (1) | Yes | Yes | Reads only; writes fail |
| ONE (1) | ALL (3) | Yes | Yes | Writes only; reads fail |
A stale read at ONE
The lab's three-node cluster turns hints off (they would otherwise hide the problem, as the previous lesson showed), stops node c3, writes items = 3 at QUORUM to c1 and c2, then brings c3 back. c3 still holds the previous value, 2. A read at ONE coordinated by c3:
Read traceRead at ONE, coordinated by c3, traced
Real database outputWritten at QUORUM (c1 and c2) while c3 was down: items = 3 Read at ONE, coordinated by c3: items ------- 2 (1 rows) Executing single-partition query on carts Read 1 live rows and 0 tombstone cells Request complete
A quorum read finds the difference, and repairs it
Read traceRead at QUORUM, coordinated by c3, traced
Real database outputitems ------- 3 (1 rows) reading digest from /c1:7000 Sending READ_REQ message to bb-c1/c1:7000 message size 91 bytes Read 1 live rows and 0 tombstone cells Read 1 live rows and 0 tombstone cells Sending READ_RSP message to /c3:7000 message size 49 bytes Digest mismatch: Mismatch for key DecoratedKey(5699955792253506986, 616c696365)
Read traceThe repair, later in the same trace
Real database outputSending read-repair-mutation to Full(/c3:7000,(5357399090202236893,5810918674640069629]) Read 0 live rows and 0 tombstone cells Sending READ_REPAIR_REQ message to /c3:7000 message size 81 bytes READ_REPAIR_REQ message received from /c3:7000 Sending READ_REPAIR_RSP message to /c3:7000 message size 32 bytes READ_REPAIR_RSP message received from /c3:7000 Request complete
read_repair = 'BLOCKING' makes the client wait for it, which the documentation says gives monotonic quorum reads: once a quorum read has returned a value, later quorum reads do not go back to an older one.Database outputRead at ONE on c3 again, after the quorum read
Real database outputRead at ONE on c3 again, after the QUORUM read: Consistency level set to ONE. items ------- 3
The price: availability
Stronger reads need more replicas alive. With two of three nodes stopped:
Database outputTwo of three replicas down
Real database outputDN c2 100.0%
UN c1 100.0%
DN c3 100.0%
Read at QUORUM:
code=1000 [Unavailable exception] message="Cannot achieve consistency level QUORUM" info={'consistency': 'QUORUM', 'required_replicas': 2, 'alive_replicas': 1}
Consistency level set to QUORUM.
Read at ONE:
Consistency level set to ONE.
items
-------
3Postgres: quorum for durability, one primary for reads
Postgres reads and writes on the primary are linearizable: there is one copy that takes writes. Quorums appear in its synchronous replication. synchronous_standby_names = 'ANY 1 (r1, r2)' makes each commit wait for any one of the two replicas, so a commit is on at least two machines, and one replica can fail without stopping writes.
Database outputQuorum commit with r1 down
Real database outputapplication_name | sync_state ------------------+------------ r2 | quorum (1 row) Time: 4.790 ms
Reads from Postgres replicas remain eventually consistent, whatever the commit setting, unless the commit used remote_apply and the read goes to a replica that applied it. Postgres has no per-read quorum: the replica you read from is the one you get.
Architecture
Drawing diagram…
Diagram. A write at quorum reaches c1 and c2, which now hold 3, and misses c3, which holds 2. A read at ONE through c3 asks only c3 and gets 2. A read at quorum through c3 asks c3 and c1; the digests differ, the newer value 3 wins and is written back to c3.
flowchart LR W["Write at QUORUM: items = 3"] --> A["c1 has 3"] W --> B["c2 has 3"] W -.->|missed, node was down| C["c3 has 2"] R1["Read at ONE via c3"] --> C R2["Read at QUORUM via c3"] --> C R2 --> A A -. "digest mismatch, then read repair" .-> C
- The write reached two replicas and the quorum read asked two: they must share at least one, here c1.
- The read at ONE asked only c3, which is in neither the write's set nor any overlap guarantee.
- Read repair makes the quorum read also heal the replica it found stale, but only for the rows it read.
Step by step
Drawing diagram…
Sequence diagram. The client reads at quorum through c3. c3 reads its local copy, 2, and asks c1 for a digest. The digests differ, so c3 requests the full data from c1, which returns 3 with a newer timestamp. c3 writes 3 to itself as read repair and returns 3 to the client.
sequenceDiagram participant App as Client participant C3 as c3 (coordinator, stale) participant C1 as c1 (replica) App->>C3: SELECT at QUORUM C3->>C3: read local copy, items = 2 C3->>C1: digest request C1-->>C3: digest of items = 3 Note over C3: digests differ C3->>C1: full data request C1-->>C3: items = 3, newer timestamp C3->>C3: read repair: write items = 3 locally C3-->>App: items = 3
- Digests keep the common case cheap: when replicas agree, only one sends the full data.
- Conflicts are resolved by write timestamp, per column. The newest timestamp wins, even if it came from a skewed clock.
- With
read_repair = 'BLOCKING'(the default), the client waits for the repair, which is what makes a later quorum read unable to return the older value.
In a real system
- Default to QUORUM for both reads and writes (LOCAL_QUORUM in multi-data-centre clusters, so requests do not wait on another region). Weaken a specific path only after stating what it may return.
- Use ONE where staleness is harmless: view counters, recommendations, logs. Write it down next to the query.
- Know your replication factor. With RF 3, QUORUM tolerates one replica down; with RF 2, QUORUM is 2 of 2 and tolerates none.
- In Postgres, use ANY 1 of two or more replicas for durable commits that survive a replica failing, and read your own writes from the primary.
- Linearizability is per operation and costly. Use it (Postgres primary, Cassandra lightweight transactions) for the few decisions that need it: uniqueness, balances, leases.
Wrong, then right
A shopping-cart service on Cassandra (RF 3) writes cart changes at ONE for speed and reads at ONE. Customers report items disappearing from their cart and reappearing.
Incorrect implementation
SimpleStatement add = SimpleStatement .newInstance("UPDATE carts SET items = items + ? WHERE user_id = ?", added, userId) .setConsistencyLevel(DefaultConsistencyLevel.ONE);SimpleStatement read = SimpleStatement .newInstance("SELECT items FROM carts WHERE user_id = ?", userId) .setConsistencyLevel(DefaultConsistencyLevel.ONE);Why it is written this way
- W = 1 and R = 1 with N = 3: 1 + 1 is not more than 3, so a read can miss the latest write.
- Each request may be coordinated by a different node and served by a different replica.
Why it fails
- A customer adds an item (acknowledged by one replica), then the cart page reads from another replica that has not received it yet: the item is gone. A refresh hits a third replica: it is back.
- If the one replica that acknowledged the write fails before the others receive it, the write can be lost entirely.
Corrected implementation
// N = 3: QUORUM (2) + QUORUM (2) > 3, so a read always overlaps the latest acknowledged write.SimpleStatement add = SimpleStatement .newInstance("UPDATE carts SET items = items + ? WHERE user_id = ?", added, userId) .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM);SimpleStatement read = SimpleStatement .newInstance("SELECT items FROM carts WHERE user_id = ?", userId) .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM);Why it is written this way
- Every acknowledged add is on two replicas, and every read asks two, so they overlap.
- LOCAL_QUORUM counts replicas in the coordinator's data centre only, so a cart request never waits on another region.
Trade-offs
- Each request now waits for the second-fastest replica instead of the fastest, which raises tail latency.
- With two of three replicas down in the data centre, cart reads and writes fail instead of returning possibly wrong data. For a cart, failing is the right choice.
- Concurrent adds from two devices are still settled per column by timestamp; a set column (
items + ?) merges them, a plain value would not.
Configuration
Where consistency is chosen. Defaults are those of PostgreSQL 18, Cassandra 5.0 and the DataStax Java driver 4.
Consistency level per request (Cassandra driver)Default LOCAL_ONE (driver default)
- What it does
- How many replicas must answer a read or acknowledge a write.
- When to change it
- Set it explicitly per statement or per profile; do not rely on the driver default for data that must be fresh.
read_repair (Cassandra table option)Default BLOCKING
- What it does
- Whether a read above ONE that finds differing replicas repairs them before answering.
- When to change it
- Leave it; NONE trades monotonic quorum reads for slightly faster reads.
replication (Cassandra keyspace)Default Set per keyspace
- What it does
- N, the number of replicas per data centre.
- When to change it
- Use NetworkTopologyStrategy with 3 per data centre for production data, so QUORUM tolerates one replica down.
synchronous_standby_names (Postgres)Default '' (none)
- What it does
- With
ANY n (list), a commit waits for any n of the listed replicas. - When to change it
- ANY 1 of two or more replicas makes every commit durable on two machines without a single replica able to stop writes.
Hands-on exercise
Make a replica stale, then catch it
Run the lab and see, on a real three-node Cassandra 5.0 cluster, a stale read at ONE, a quorum read detecting it with a digest mismatch and repairing it, and QUORUM refusing to answer when two of three replicas are down; and Postgres committing with a quorum of replicas while one is down.
Steps · 0 of 5 done
What you should see
The read at ONE returns the old value 2 with only a local read in its trace. The QUORUM read reports a digest mismatch, returns 3 and sends a read-repair mutation, after which ONE on the same node returns 3. With two nodes down, QUORUM is unavailable (2 required, 1 alive) and ONE answers. In Postgres, with ANY 1 (r1, r2) and r1 down, r2 shows as quorum and the insert commits in milliseconds. A write at ALL plus a read at ONE satisfies 3 + 1 > 3, so that read cannot be stale.
What can go wrong
QUORUM with replication factor 2
- Symptom
- Every request fails while one node is restarted for an upgrade.
- Cause
- QUORUM of 2 is 2: it needs both replicas.
- Fix
- Use replication factor 3 for data read or written at QUORUM.
QUORUM across data centres
- Symptom
- Latency jumps by a cross-region round trip, and requests fail when the link between regions is slow.
- Cause
- QUORUM counts replicas in all data centres, so it often has to wait for a remote one.
- Fix
- Use LOCAL_QUORUM, and accept that another region may briefly see older data.
Assuming a failed write was not applied
- Symptom
- A write that timed out at QUORUM shows up later.
- Cause
- It reached fewer replicas than required before the timeout but was not rolled back; repair spreads it.
- Fix
- Treat timeouts as unknown, not failed; make writes idempotent so a retry is harmless.
Last write wins with skewed clocks
- Symptom
- An older update overwrites a newer one.
- Cause
- Conflicts are settled by write timestamp, set by the client or coordinator clock; a clock ahead wins.
- Fix
- Keep clocks synchronised (NTP), and use lightweight transactions where the order of updates matters.
How to troubleshoot
Clients see Unavailable errors
Check
nodetool status <keyspace>
- How to read it
- The error's
required_replicasandalive_replicastell you how many replicas were down. Compare with the keyspace's replication factor and the consistency level used.
Users see old values intermittently in Cassandra
Check
Trace a read at the level the application uses (cqlsh: CONSISTENCY ONE; TRACING ON; SELECT ...)
- How to read it
- A read at ONE that touches one replica can be stale; digest mismatches in quorum traces show replicas are diverging, which points at missed hints or overdue repair.
Postgres commits slow down or hang when one replica has a problem
Check
SHOW synchronous_standby_names; SELECT application_name, sync_state FROM pg_stat_replication;
- How to read it
- A single name, or FIRST 1 with only one healthy listed replica, makes that replica a dependency of every commit.
Key takeaways
- Consistency models range from eventual to linearizable; name the one each read path gives.
- In Cassandra, reads see the latest acknowledged write when R + W > N; QUORUM plus QUORUM with N = 3 is the usual choice.
- A read at ONE can silently return an old value; a quorum read detects the difference with digests and repairs it.
- Stronger levels fail sooner when replicas are down: with two of three down, QUORUM is unavailable and ONE still answers.
- Postgres is linearizable on the primary; ANY n quorum commit makes writes durable on several replicas without one replica stopping commits.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A Cassandra keyspace has replication factor 3. A service writes user settings at ONE and reads them at QUORUM. A user changes a setting and reloads.
Is the reload guaranteed to show the change?
Scenario 2 of 3
A payments ledger on Cassandra uses replication factor 3 in each of two data centres and writes at QUORUM. Write latency doubled after the second data centre was added, and spikes whenever the link between them is slow.
What change keeps fresh local reads and removes the cross-region wait?
Scenario 3 of 3
A Postgres primary has two replicas. The team wants every commit to be on at least two machines, but a single replica failing must not stop writes.
Which setting achieves that?
References
References: Apache Cassandra 5.0 documentation: Dynamo (tunable consistency, R + W > N) ↗ · Apache Cassandra 5.0 documentation: Read repair (BLOCKING and monotonic quorum reads) ↗ · Apache Cassandra 5.0 documentation: Guarantees ↗ · Apache Cassandra 5.0 documentation: Repair ↗ · PostgreSQL 18 documentation: Replication settings (synchronous_standby_names, ANY and FIRST) ↗ · DeCandia et al., Dynamo: Amazon's Highly Available Key-value Store (SOSP 2007) ↗ · Jepsen: Consistency models ↗ · Herlihy and Wing, Linearizability: A Correctness Condition for Concurrent Objects (1990) ↗ · Report a problem