Skip to content
Foundation15 minSimulation 2 · Java 21

Lost updates: counter++ is three steps

Step two threads through read, add, write on one shared counter and lose updates on purpose. Then see why volatile does not help and why a lock or a compare-and-set does.

The situation

A Spring Boot 3 promo service keeps two numbers in a singleton bean: how many vouchers were redeemed, and how many of the 500 vouchers of a campaign are left. Every request runs on one of Tomcat's worker threads (200 by default), and all of them call the same bean. After a busy evening the dashboard reads 7,412 redemptions while the orders table has 7,950 rows, and 506 customers hold one of the 500 vouchers. Nothing was logged, no exception was thrown, and the numbers are right in every test that runs on one thread. Someone already made the field volatile; the gap got smaller and did not close.

Read first: Race conditions: visibility and atomicity, synchronized, volatile and happens-before, Atomics and compare-and-set

Break it in the simulation

Be the scheduler for counter++

Simulation · not a real JVM

Two threads, three increments each, expected result 6. Press Step T1 once, then let T2 finish an increment, then go back to T1 and watch its WRITE wipe out T2's work. Worst case does the most damage possible.

Guided run · 0 of 6 stages done

    1. Leave How the counter is protected on Plain int, Threads at 2 and Increments per thread at 3.
    2. Press Lucky run.

    Look for

    • counter is 6, Expected is 6 and Lost updates is 0.
    • The verdict says nothing was lost, by luck. T1 ran all 9 of its steps before T2 started. This is what a single-threaded test sees.
    1. Press Reset.
    2. Press Step T1 once. T1 reads 0.
    3. Press Step T2 three times: READ, ADD, WRITE. T2 has finished one increment and counter is 1.
    4. Press Step T1 twice: ADD, then WRITE.

    Look for

    • Before T1 writes, its card shows Value it read: 0 (stale, counter is 1). The register holds 1, computed from that 0.
    • After T1's WRITE, counter is still 1 but Increments finished says 2 of 6. Lost updates is 1.
    • In Steps taken, step 6 is tagged WRITE LOST: T1 wrote a value computed from an old READ on top of T2's increment.
    1. Press Worst case.
    2. Read Steps taken from the top. Look at step 9 and step 18.

    Look for

    • counter ends at 2, Expected is 6: Lost updates is 4, and the verdict says 66.7% lost.
    • Step 9: T1 writes 1, computed from the 0 it read in step 1, over a counter that had reached 2.
    • Step 18: T2 writes 2, computed from the 1 it read in step 10, over a counter that T1 had brought to 3. Two stale writes are enough to throw away four increments.
    • Set Threads to 4 and Increments per thread to 5 and press Worst case again: the counter still ends at 2, now out of 20.
    1. Set Threads back to 2 and Increments per thread to 3.
    2. Choose volatile int.
    3. Press Worst case.

    Look for

    • The same 18 steps, the same result: counter 2, Lost updates 4.
    • Every READ in Steps taken did return the latest value. The value went stale afterwards, between READ and WRITE. volatile says nothing about that gap.
    • The verdict quotes the real lab: the volatile int lost 40.6% of 1,600,000 increments.
    1. Choose synchronized.
    2. Press Step T1 once. T1 takes the lock.
    3. Look at T2's card, then press Worst case.

    Look for

    • T2's status is BLOCKED and its Step T2 button is disabled until T1 has pressed through READ, ADD, WRITE and UNLOCK. Lock reads held by T1.
    • After Worst case: counter 6 of 6, Lost updates 0. The threads took turns at every opportunity and still could not interleave inside an increment.
    • It took 30 steps instead of 18: each increment pays for a LOCK and an UNLOCK, and a blocked thread does nothing while it waits.
    1. Choose AtomicInteger.
    2. Press Worst case.
    3. Find the lines tagged CAS RETRY in Steps taken.

    Look for

    • counter 6 of 6, Lost updates 0, CAS retries 3.
    • Both threads read the same value. The first compare-and-set succeeds. The second one finds that the counter is no longer the value it read, writes nothing, and goes back to READ.
    • No thread is ever BLOCKED. The cost of contention is repeated work, 27 steps instead of 18, not waiting.

Wrong, then right

Incorrect implementation

src/main/java/com/example/promo/RedemptionStats.javaJava
package com.example.promo;import java.util.HashMap;import java.util.Map;import org.springframework.stereotype.Service;/** A singleton bean: every Tomcat request thread calls the same instance. */@Servicepublic class RedemptionStats {    // "volatile" was added after the first bug report. It did not help.    private volatile long redeemed;    private int remaining = 500;    private final Map<String, Long> perCampaign = new HashMap<>();    public void recordRedemption(String campaign) {        redeemed++;                                   // read, add, write        perCampaign.merge(campaign, 1L, Long::sum);   // HashMap: same race, plus a corrupted table    }    /** Hands out one of 500 vouchers. */    public boolean tryReserve() {        if (remaining > 0) {      // check            remaining--;          // then act: two threads can both pass the check            return true;        }        return false;    }    public long redeemed() {        return redeemed;    }}

Why it is written this way

  • redeemed++ on a field is a read, an add and a write. Two request threads that read the same value both write value + 1, and one redemption disappears.
  • volatile makes each read return the latest write. It does not make the three steps one, so the counter still loses updates. The lab measured 40.6% lost with a volatile int.
  • tryReserve is the same bug in its check-then-act form: two threads can both see remaining == 1, both pass the check and both decrement. That is how 506 customers get 500 vouchers.
  • HashMap.merge from several threads loses counts the same way, and concurrent writes can also damage the map's internal table.

Why it fails

  • A @Service bean is a singleton by default, so its fields are shared memory for every request thread. Nothing in Spring makes access to them atomic.
  • redeemed++ is not one operation. The thread reads the field, adds 1 in its own register and writes the result back. If another thread writes between the read and the write, that write is overwritten: the counter goes up by 1 for two increments.
  • The damage is not limited to one update per collision. A thread that is descheduled between its read and its write can come back much later and write its old value + 1 over many increments made in the meantime. In the widget's worst case two such writes reduce any number of increments to 2.
  • volatile fixes visibility, not atomicity. The Java Language Specification guarantees that a write to a volatile field happens-before every subsequent read of it. It says nothing that joins a read and a later write of the same field.
  • Nothing fails loudly. There is no exception and no log line, and a test with one thread always passes. On the real JVM the lab lost 54.1% of 1,600,000 plain increments and 40.6% with volatile, in under 20 ms each.

Corrected implementation

src/main/java/com/example/promo/RedemptionStats.javaJava
package com.example.promo;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.atomic.AtomicInteger;import java.util.concurrent.atomic.LongAdder;import java.util.stream.Collectors;import org.springframework.stereotype.Service;@Servicepublic class RedemptionStats {    private final LongAdder redeemed = new LongAdder();    private final AtomicInteger remaining = new AtomicInteger(500);    private final ConcurrentHashMap<String, LongAdder> perCampaign = new ConcurrentHashMap<>();    public void recordRedemption(String campaign) {        redeemed.increment();        perCampaign.computeIfAbsent(campaign, c -> new LongAdder()).increment();    }    /** Hands out one of 500 vouchers. Check and decrement are one compare-and-set. */    public boolean tryReserve() {        while (true) {            int current = remaining.get();            if (current == 0) {                return false;            }            if (remaining.compareAndSet(current, current - 1)) {                return true;            }            // Another thread changed the value between get() and compareAndSet(): read again.        }    }    public long redeemed() {        return redeemed.sum();    }    public Map<String, Long> perCampaign() {        return perCampaign.entrySet().stream()                .collect(Collectors.toMap(Map.Entry::getKey, e -> e.getValue().sum()));    }}

Why it is written this way

  • LongAdder for a number that is written often and read rarely. increment() never loses an update.
  • AtomicInteger with a compare-and-set loop where the value decides something. The check (current == 0) and the decrement are tied together: the write happens only if the value is still the one that was checked.
  • ConcurrentHashMap.computeIfAbsent creates each per-campaign counter once, atomically, and the LongAdder inside takes the increments.
  • The fields are final: the references never change, only the state inside the thread-safe objects.
  • Not compiled against Spring here: the class was compiled with JDK 21 using a stub for @Service.

Trade-offs

  • LongAdder.sum() is not an atomic snapshot: increments that happen while it adds up its cells may or may not be included. Fine for a dashboard, wrong for a decision such as "is the limit reached".
  • LongAdder uses more memory than AtomicLong under contention, in exchange for higher throughput when many threads update it. Under low contention the two behave alike.
  • A CAS loop repeats work under contention: every failed compareAndSet costs another read and compute. The function inside the loop must have no side effects, because it can run more than once.
  • An atomic covers one variable. If two fields must change together (the voucher count and the list of holders), you need a lock around both, or one immutable object swapped with a single compare-and-set.
  • synchronized is the simplest correct fix and the right one when several fields are involved. Its cost is that threads queue: in the lab it took 34 ms where the plain int took 19 ms, on 2 CPUs.
  • All of this is per JVM. With three pods there are three counters. A voucher limit that must hold across instances belongs in the database (a conditional UPDATE or a unique constraint), not in a bean field.

Run it on a real JVM

LostUpdate.java starts 8 threads behind a latch and lets each add 1 to a shared counter 200,000 times, five times over: a plain int, a volatile int, an int guarded by a synchronized block, an AtomicInteger and a LongAdder. It prints the result, the number of lost updates and the time for each. Running it proves that the first two lose a large share of the updates and that the other three lose none.

With Docker (CPU and memory limits)

labs/jvm/lost-updatesShell
cd labs/jvm/lost-updatesdocker build -t jvm-lab-lost-updates .# all five variants in one run: the two broken ones first, then the three fixesdocker run --rm --cpus=2 --memory=512m jvm-lab-lost-updates LostUpdate# run it a few times: the lost counts change on every run, the three zeros do notdocker run --rm --cpus=2 --memory=512m jvm-lab-lost-updates LostUpdate# the same with one CPU: compare the lost counts with the 2 CPU rundocker run --rm --cpus=1 --memory=512m jvm-lab-lost-updates LostUpdate

Without Docker, on a local JDK 21

labs/jvm/lost-updatesShell
cd labs/jvm/lost-updatesjava -XX:ActiveProcessorCount=2 LostUpdate.java# there is nothing to attach to: the program ends in well under a second
The program: LostUpdate.java
labs/jvm/lost-updates/LostUpdate.javaJava
import java.util.concurrent.CountDownLatch;import java.util.concurrent.atomic.AtomicInteger;import java.util.concurrent.atomic.LongAdder;/** * Lab 2: race conditions and lost updates. * * Eight threads each add 1 to a shared counter 200,000 times. The right answer is 1,600,000. * Run:  java LostUpdate.java */public class LostUpdate {    static final int THREADS = 8;    static final int PER_THREAD = 200_000;    static int plain;                 // no protection at all    static volatile int volatileInt;  // visible to every thread, but ++ is still three steps    static int guarded;               // protected by a lock    static final Object lock = new Object();    static final AtomicInteger atomic = new AtomicInteger();    static final LongAdder adder = new LongAdder();    public static void main(String[] args) throws Exception {        int expected = THREADS * PER_THREAD;        System.out.printf("%d threads x %,d increments, expected %,d%n%n", THREADS, PER_THREAD, expected);        report("int, no synchronisation", run(() -> plain++), plain, expected);        report("volatile int", run(() -> volatileInt++), volatileInt, expected);        report("synchronized block", run(() -> { synchronized (lock) { guarded++; } }), guarded, expected);        report("AtomicInteger.incrementAndGet", run(atomic::incrementAndGet), atomic.get(), expected);        report("LongAdder.increment", run(adder::increment), adder.sum(), expected);    }    static long run(Runnable increment) throws InterruptedException {        CountDownLatch start = new CountDownLatch(1);        Thread[] threads = new Thread[THREADS];        for (int t = 0; t < THREADS; t++) {            threads[t] = new Thread(() -> {                try { start.await(); } catch (InterruptedException e) { return; }                for (int i = 0; i < PER_THREAD; i++) increment.run();            });            threads[t].start();        }        long t0 = System.nanoTime();        start.countDown();        for (Thread t : threads) t.join();        return (System.nanoTime() - t0) / 1_000_000;    }    static void report(String name, long ms, long got, long expected) {        long lost = expected - got;        System.out.printf("%-32s result %,9d   lost %,9d (%4.1f%%)   %4d ms%n", name, got, lost, 100.0 * lost / expected, ms);    }}
The Dockerfile
labs/jvm/lost-updates/DockerfileDockerfile
# Java 21 lab image. Build:  docker build -t jvm-lab-lost-updates .FROM eclipse-temurin:21-jdkWORKDIR /labCOPY LostUpdate.java .RUN javac -d out LostUpdate.java# The JVM reads the container's CPU and memory limits (container support is on by default),# so --cpus and --memory on "docker run" change what Runtime.availableProcessors() and the default heap size report.ENTRYPOINT ["java", "-cp", "out"]CMD ["LostUpdate"]

$ java -XX:ActiveProcessorCount=2 LostUpdate.java

Program outputFive ways to add 1, 1,600,000 times

Real JVM output
8 threads x 200,000 increments, expected 1,600,000

int, no synchronisation          result   734,295   lost   865,705 (54.1%)     19 ms
volatile int                     result   951,024   lost   648,976 (40.6%)     15 ms
synchronized block               result 1,600,000   lost         0 ( 0.0%)     34 ms
AtomicInteger.incrementAndGet    result 1,600,000   lost         0 ( 0.0%)     10 ms
LongAdder.increment              result 1,600,000   lost         0 ( 0.0%)     18 ms
Read the lost column. The plain int ended at 734,295 of 1,600,000: 54.1% of the increments are gone. The volatile int still lost 40.6%. The synchronized block, AtomicInteger and LongAdder lost 0. The last column is the price: synchronized took 34 ms against 10 ms for AtomicInteger in this run. These are single runs of a few milliseconds, so do not read much into the exact times.

Real run and simulation: what each one shows

  • The captured file comes from OpenJDK 21.0.13 run directly on Linux x86-64 with -XX:ActiveProcessorCount=2, not in Docker. The Docker commands above were not executed when this was written.
  • The real run shows how often it happens in practice: with 8 threads on 2 CPUs more than half of all plain increments were lost within 19 ms. The widget cannot tell you a rate, only what is possible.
  • The real run also includes effects the model leaves out, such as the JIT compiler keeping a plain field in a register. One run cannot separate those causes, and the percentages change from run to run.
  • The simulation shows what the real run cannot: the exact interleaving behind a lost update, one step at a time, and the worst case (a final value of 2), which a real scheduler is very unlikely to produce.
  • The lab has no line for a failed compare-and-set. The widget's CAS retries show the mechanism; the real AtomicInteger just reports the correct total.

How you meet this in production

  • The first symptom is a number that is slightly too low and only under load: a counter that disagrees with the database, a metric that drifts below the request count, a limit that is exceeded by a few. It never reproduces on a developer machine with one request at a time.
  • No tool points at it directly. A thread dump shows nothing, because no thread is stuck; there is no exception and no GC sign. You find it by reading code: look for mutable fields in singleton beans (@Service, @Component, @RestController) that are written with ++, +=, or check-then-act, without a lock or an atomic.
  • To confirm, write a test like the lab: N threads behind a CountDownLatch, many iterations, compare with the expected total. One failing run proves the race. A passing run proves nothing.
  • volatile on a counter is a warning sign in review. It usually means someone saw the bug, fixed visibility and left atomicity broken.

References

References: Java Language Specification, Java SE 21, chapter 17: Threads and Locks (17.4.5 happens-before) ↗ · AtomicInteger (Java SE 21 API) ↗ · LongAdder (Java SE 21 API) ↗ · OpenJDK 21 source: jdk.internal.misc.Unsafe (getAndAddInt) ↗ · Spring Boot: common application properties (server.tomcat.threads.max) ↗ · Report a problem