Incident · Hard · Databases
Two leaders, no partition
- SEV2
- SEARCH CLUSTER
- DATA DIVERGED
90s
The situation
Your search cluster ends up with two nodes that each believe they are the primary. Both accept writes, and the data diverges. The network team checks and swears there was no partition: every link stayed up the whole time.
The evidence
Logs and signalsReconstructed
02:14:07node-1 (primary): GC pause started02:14:21node-2, node-3: primary not responding02:14:22node-2: elected primary (min votes required: 1)02:14:25node-1: GC pause ended (18s) · still primary02:14:30network: 0 packets lost
Your call
How did you get two primaries?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.