Incident · Medium · URL shortener · Databases
#7, then #39
- NO DOWNTIME
- CUSTOMER REPORTS
- IDS JUMPED
90s
The situation
Your product gives each customer its own run of friendly reference numbers: #1, #2, #3. Each run is a database sequence. Customers start writing in. One with 20 records has just been handed #52. Another went from #7 straight to #39. No rows are missing, nothing errored and the database has not crashed. It was upgraded recently, by building a follower and promoting it.
The evidence
- number issued
- records that exist
Logs and signalsReconstructed
-- customer A: 20 records, newest is #52-- customer B: #7 then #39-- audit: gap is exactly 32 on many sequences-- SELECT * FROM seq → last_value 186 · log_cnt 32-- failed or rolled-back creates: none found-- last week: database upgraded (follower promoted)
Your call
Where did the 32 numbers go?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.