Skip to content
All outages

Incident · Hard · Databases

Two leaders, no partition

  • SEV2
  • SEARCH CLUSTER
  • DATA DIVERGED
90s

The situation

Your search cluster ends up with two nodes that each believe they are the primary. Both accept writes, and the data diverges. The network team checks and swears there was no partition: every link stayed up the whole time.

The evidence

Logs and signalsReconstructed
02:14:07node-1 (primary): GC pause started
02:14:21node-2, node-3: primary not responding
02:14:22node-2: elected primary (min votes required: 1)
02:14:25node-1: GC pause ended (18s) · still primary
02:14:30network: 0 packets lost

Your call

How did you get two primaries?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Third-party account