Skip to content

Kafka / streaming · 3 of 9

Step 1 · The pattern

A log between two speeds

The producer writes messages to a topic. A consumer reads each one and writes it to the database. The topic sits between them so neither has to wait for the other.

  • message
  • processed

In short

Consumer lag & backpressure: how it works and how it breaks

Consumer lag is the distance between the newest message in a partition and the last one a consumer has finished. A little lag is normal. It is the point of a log: producers can run ahead and the consumer catches up later. Lag becomes an outage when the consumer handles a burst by doing more work at once. It pulls messages as fast as it can and fires them at a database or API with a fixed capacity. That dependency slows under the pile-up, requests time out and are redelivered, and each message now costs more than before. Throughput drops below the normal arrival rate, so the backlog grows even after the burst ends. Backpressure is the cure. Limit work in flight to what the downstream system handles comfortably and leave the rest in the log, where waiting is cheap.

How this comes up in interviews

"Consumer lag keeps growing even though the traffic burst ended an hour ago. Walk me through how you investigate and fix it."

Then they ask

  1. The database is at full CPU and someone proposes doubling the consumers. What happens if you do?
  2. How do you decide the limit on work in flight, and where do the waiting messages live?
  3. Which metrics tell you a backlog is draining and not turning into an outage?

A strong answer covers

  • Lag is the gap between the newest message and the last one finished. Some lag is normal if it drains.
  • Pushing everything at a fixed-capacity dependency slows it, causes timeouts and redelivery, and drops throughput.
  • Once throughput falls below the normal arrival rate, the backlog grows after the burst is over.
  • Find the real bottleneck first. If it is downstream, more consumers add concurrent work and make it worse.
  • Backpressure: cap work in flight at what the downstream system handles well and leave the rest in the log.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.