Incident · Medium · Redis / caching
Someone else's chat titles
- SEV1
- PRIVACY
- PRODUCT TAKEN OFFLINE
90s
The situation
Users of a chat product open their sidebar and see conversation titles they never wrote. A few get a subscription email meant for a stranger. The user cache is a healthy Redis Cluster, and every key you inspect holds the right user's data. Server errors have been elevated since a server change at 01:00.
The evidence
- cancelled requests
- mismatched replies
Logs and signalsReconstructed
01:00deploy: server change live01:02cache client: request cancelled before reply was read01:02conn #41 returned to pool01:02conn #41: unexpected reply type → 50001:03GET user:8841 on conn #17 → record for user:229001:03redis: user:8841 holds user 8841 (correct)
Your call
How do the replies get crossed?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.