Skip to content
You're viewing documentation for v2026.11.5-beta. It may be out of date.Go to latest
TimescaleDB Enterprise
Esc
↑↓navigate↵open⌘Jpreview
On this page

Node power loss

Bring back a PostgreSQL server of an HA service after a power loss, confirm that it rejoins as a replica, and check for lost commits.

What happens

  • The primary loses power: after 30 s, Patroni promotes a replica. The other replicas follow it.
  • The server holds the VIP: the VIP moves to a surviving server. Connections through the VIP drop once. Clients with target_session_attrs=read-write then reach the new primary.
  • A replica loses power: the primary keeps serving.

Replication is asynchronous. A commit that the old primary acknowledged but the new primary did not receive is lost.

Symptoms

  • Console: on the service Overview, Database service health shows Patroni needs attention during the failover. Then the topology shows a different leader. On Infrastructure → Servers, the Health of the server is Unreachable, and Test server shows SSH connection failed.
  • Clients: connections through the VIP fail for some seconds. Writes in progress at the power loss return an error. The application must retry them.

Recovery

Repair the hardware

Repair the hardware fault. If the server is lost, use Replace a server.

Recover the console

If the console is down, recover it first (see A node restarts during the outage). The server must reach the console to join the mesh again.

Power on the server

Power on the server. Patroni starts at boot and rejoins the cluster.

Wait for the server

In the topology on the service Overview, wait for the server:

  • Good: the server shows a healthy status dot and no state label. A replica shows a lag near 0.
  • Bad: there is no leader. Contact support. Do not rebuild a member while there is no leader.

Find the cause

If the member stays in start failed or crashed, or its lag does not go down, find the cause. If you cannot find it, contact support.

Rebuild the member

If the member needs a rebuild, see Rebuild a member.

Test the server

On Infrastructure → Servers, select Test server. It must show SSH connection established. If not, contact support.

Rebuild a member

The console has no button for a rebuild. Use the API (admin): POST /api/v1/database-services/{id}/servers/{server_id}/rebuild-replica. See the API reference. The console refuses the primary, a server that it cannot reach, and a rebuild while another server is unhealthy.

Wait until the member shows streaming with a lag near 0 in the topology. If the rebuild fails or stops, contact support.

If the lost server was the primary

Commits from the last moments before the power loss can be missing.

  • Applications that retry writes: find the rows that the application wrote but cannot read back. Reconcile them with the idempotency rules of the application before you write them again.
  • Applications that do not retry: compare the last committed record with the source of the data, for example a message queue offset.

Prevention

  • Put the HA servers on different hosts and different power circuits.
  • Keep a backup server attached. A restore recovers a lost server.
  • Configure alert delivery. PatroniNoLeader fires when a cluster has no leader for 5 minutes. When the Patroni endpoints do not answer, ServiceDown fires instead.