Skip to content
All outages

Editor's note: No downtime: a surprise customers noticed.

Incident · Medium · URL shortener · Databases

#7, then #39

  • NO DOWNTIME
  • CUSTOMER REPORTS
  • IDS JUMPED
90s

The situation

Your product gives each customer its own run of friendly reference numbers: #1, #2, #3. Each run is a database sequence. Customers start writing in. One with 20 records has just been handed #52. Another went from #7 straight to #39. No rows are missing, nothing errored and the database has not crashed. It was upgraded recently, by building a follower and promoting it.

The evidence

One customer's reference numbers, in order createdillustrative
  • number issued
  • records that exist
Logs and signalsReconstructed
-- customer A: 20 records, newest is #52
-- customer B: #7 then #39
-- audit: gap is exactly 32 on many sequences
-- SELECT * FROM seq → last_value 186 · log_cnt 32
-- failed or rolled-back creates: none found
-- last week: database upgraded (follower promoted)

Your call

Where did the 32 numbers go?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Engineering write-up