Network partition
Bring back an HA server that lost its network link, confirm that it rejoins as a replica, and check that the service has one primary only.
What happens
- The isolated server is a replica. The primary stays. If the replica held the VIP, the VIP moves. Clients of that replica get errors.
- The isolated server is the primary. If it still reaches all other members, it stays the primary. If not, Patroni demotes it, or the watchdog resets the server after about 25 s. The other members can promote a replica. If the primary held the VIP, the VIP moves, and connections through the VIP fail during the move.
- When the link returns, the old primary rejoins as a replica.
Symptoms
- On Infrastructure → Mesh, the server shows Unreachable.
- On the service Overview, Database service health shows Patroni needs attention.
- If the isolated server was the primary, connections through the VIP fail for a short time. Then a different member is the leader.
Recovery
Repair the link
Repair the link: the cable, the switch port or the interface.
Wait for the server
In the topology on the service Overview, wait for the server to rejoin.
- Good: one member is the leader, and the server is a replica with a lag near 0. If the server stayed the leader, the other members are replicas.
- Bad: there is no leader. Contact support. Do not rebuild a member.
- Bad: the server does not rejoin. Go to If the server does not rejoin.
Check the mesh
On Infrastructure → Mesh, confirm that the server shows Online. If it stays Unreachable, check its network. Contact support if the cause is not clear.
Check the service health
On the service Overview, confirm that Database service health is healthy. If the server was the primary, check the application records for missing commits. See If the lost server was the primary.
If the server does not rejoin
The server stays in stopped or start failed after the link is back. Find
the cause first. Contact support if it is not clear.
If the member needs a rebuild, the server must show Online on Infrastructure → Mesh. Then follow Rebuild a member. A rebuild deletes the data directory of the member, including commits that are not on the leader.
When the server shows streaming, do steps 3 and 4 of
Recovery.
Prevention
- Give each HA server redundant network paths, for example bonded NICs or two switch uplinks.
- On an Advanced HA service, put the DCS servers on network paths that are different from the PostgreSQL servers.