---
title: Node power loss
description: "Bring back a PostgreSQL server of an HA service after a power loss, confirm that it rejoins as a replica, and check for lost commits."
related:
  - /operations/site-outage
  - /operations/network-partition
---

## What happens

- **The primary loses power:** after 30 s, Patroni promotes a replica. The
  other replicas follow it.
- **The server holds the VIP:** the VIP moves to a surviving server.
  Connections through the VIP drop once. Clients with
  `target_session_attrs=read-write` then reach the new primary.
- **A replica loses power:** the primary keeps serving.

Replication is asynchronous. A commit that the old primary acknowledged but
the new primary did not receive is lost.

## Symptoms

- **Console:** on the service **Overview**, **Database service health**
  shows *Patroni needs attention* during the failover. Then the topology
  shows a different leader. On **Infrastructure → Servers**, the **Health**
  of the server is **Unreachable**, and **Test server** shows **SSH
  connection failed**.
- **Clients:** connections through the VIP fail for some seconds. Writes in
  progress at the power loss return an error. The application must retry
  them.

## Recovery

:::warning
If you need commits that exist only on the old primary, contact support
before you power it on. When it rejoins, Patroni can rewind or replace its
data directory.
:::

1. **Repair the hardware**

    Repair the hardware fault. If the server is lost, use
    [Replace a server](/docs/operations/replace-a-server).

2. **Recover the console**

    If the console is down, recover it first (see
    [A node restarts during the outage](/docs/operations/console-outage#a-node-restarts-during-the-outage)).
    The server must reach the console to join the mesh again.

3. **Power on the server**

    Power on the server. Patroni starts at boot and rejoins the cluster.

4. **Wait for the server**

    In the topology on the service **Overview**, wait for the server:

    - Good: the server shows a healthy status dot and no state label. A replica
      shows a lag near 0.
    - Bad: there is no leader. Contact support. Do not rebuild a member while
      there is no leader.

5. **Find the cause**

    If the member stays in `start failed` or `crashed`, or its lag does not
    go down, find the cause. If you cannot find it, contact support.

6. **Rebuild the member**

    If the member needs a rebuild, see [Rebuild a member](#rebuild-a-member).

7. **Test the server**

    On **Infrastructure → Servers**, select **Test server**. It must show
    **SSH connection established**. If not, contact support.

## Rebuild a member

:::warning
A rebuild deletes the data directory of the member, including commits that
are not on the leader. Then it copies the data again from the leader. First
confirm that the leader has the data that you need.
:::

The console has no button for a rebuild. Use the API (admin):
`POST /api/v1/database-services/{id}/servers/{server_id}/rebuild-replica`.
See the [API reference](/docs/reference). The console refuses the primary, a
server that it cannot reach, and a rebuild while another server is
unhealthy.

Wait until the member shows `streaming` with a lag near 0 in the topology.
If the rebuild fails or stops, contact support.

## If the lost server was the primary

Commits from the last moments before the power loss can be missing.

- **Applications that retry writes:** find the rows that the application
  wrote but cannot read back. Reconcile them with the idempotency rules of
  the application before you write them again.
- **Applications that do not retry:** compare the last committed record with
  the source of the data, for example a message queue offset.

## Prevention

- Put the HA servers on different hosts and different power circuits.
- Keep a [backup server](/docs/backups) attached. A restore recovers a lost
  server.
- Configure [alert delivery](/docs/alerts). `PatroniNoLeader` fires when a
  cluster has no leader for 5 minutes. When the Patroni endpoints do not
  answer, `ServiceDown` fires instead.
