Skip to content
All outages

Incident · Easy · Databases

One sale, every store

  • SEV1
  • SALE DAY
  • ONE SHARD
90s

The situation

You run a platform that hosts online stores, many to a database shard. A celebrity make-up brand announces a sale and fans rush checkout the moment it opens. Her store goes down, and so does every other store on its shard. You have just added heavier page caching; it helped, though not enough. Her next sale is in a week.

The evidence

Checkout writes on the shard: attempted vs completedillustrative
  • attempted
  • completed
Logs and signalsReconstructed
sale announced, new products go live
page cache: hit rate high
checkout: one record per session, rewritten at every step
shard: write load past its limit
other stores on the shard: checkout failing

Your call

What do you ship before next week's sale?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Engineering write-up