---
title: Site outage and recovery
description: "Bring back an HA service after all of its servers lost power at the same time, confirm that it has one primary, and check for lost commits."
related:
  - /operations/node-power-loss
  - /operations/network-partition
---

## What happens

All servers of the HA service stop at the same time, for example after a
power loss or a storage failure. When they start again, the service
recovers by itself. A leader is back in about 40 s, and the service is
healthy in about one minute.

A server that starts alone does not promote. On an Advanced HA service,
PostgreSQL stays stopped until the DCS servers have quorum.

## Symptoms

- Clients cannot connect through the VIP or to any server. Connections are
  refused or time out.
- On the service **Overview**, **Database service health** shows
  *etcd, Patroni, and HAProxy need attention*.

## Recovery

:::warning
If you need data that can exist only on a former primary, contact support
before you start that server. When it rejoins, Patroni can rewind or
replace its data directory.
:::

1. **Recover the console**

    If the console is down, recover it first with
    [Console outage](/docs/v2026.11.5-beta/operations/console-outage#recovery). The servers cannot
    join the mesh until the console is available.

2. **Power on all servers**

    Power on all servers. On Advanced HA, start the DCS servers first. If a
    server cannot start, do the next checks on the servers that run.

3. **Confirm etcd quorum**

    In the topology on the service **Overview**, etcd must be healthy on at
    least two nodes. On Advanced HA, the cluster on
    **Infrastructure → DCS clusters** must not show **Quorum lost**.

    If the console cannot show it, run this on a running DCS member (on
    Standard HA, a PostgreSQL server):

    ```bash
    sudo /usr/local/bin/etcdctl --endpoints=http://127.0.0.1:2379 endpoint health --cluster
    ```

    - Good: at least two members show `is healthy`.
    - Bad: fewer than two, or the command fails. Go to
      [etcd has no quorum](#etcd-has-no-quorum).

4. **Confirm the Patroni cluster**

    In the topology, one member must be the leader, and each other member a
    replica with a lag near 0.

    If the console cannot show it, run this on a running PostgreSQL server:

    ```bash
    sudo -u postgres patronictl -c /etc/patroni/patroni.yml list
    ```

    - Good: one member is `Leader`, and each other member is `streaming`.
    - Bad: no leader, or a replica does not rejoin. Go to
      [Patroni does not elect a leader or a replica does not rejoin](#patroni-does-not-elect-a-leader-or-a-replica-does-not-rejoin).

5. **Check the service health**

    Confirm that **Database service health** is healthy.

6. **Connect from a client**

    From a client, connect to port 5000 of the VIP (or of the node endpoints
    without a VIP), and confirm that you reach the leader. See
    [Connect](/docs/v2026.11.5-beta/database-services/connect). If the VIP does not answer, run
    `sudo systemctl status keepalived haproxy` on each PostgreSQL server, and
    read the log of a failed unit.

7. **Replace a lost server**

    If a server is lost for good, use
    [Replace a server](/docs/v2026.11.5-beta/operations/replace-a-server).

If a step does not recover, contact support.

## If it does not heal

### etcd has no quorum

Patroni cannot elect a leader while fewer than 2 of 3 etcd members are
healthy.

1. **Start the etcd members**

    Start each etcd member that is down. Do not remove or replace members by
    hand.

2. **Read the etcd log**

    If a member does not start, read its log on that server:
    `sudo journalctl -u etcd --since "1 hour ago"`.

3. **Continue the recovery**

    When two members are healthy, continue with step 4 of
    [Recovery](#recovery).

If a member cannot come back:

- **One member is lost with its server, and the other two are healthy:** use
  [Replace a server](/docs/v2026.11.5-beta/operations/replace-a-server).
- **Two members are lost, or a member does not start for another reason:**
  create a [support report](/docs/v2026.11.5-beta/logs-and-support#support-reports) and open a
  support case.

### Patroni does not elect a leader or a replica does not rejoin

Read the Patroni log of the server: on **Logs & Diagnostics**, select the
server and the log source **Patroni**. If the console cannot show it, run
`sudo tail -n 200 /var/log/patroni/patroni.log` on the server.

- **PostgreSQL is stopped and the log shows etcd errors.** Restore etcd
  quorum first. Patroni starts PostgreSQL when etcd answers.
- **No leader while etcd is healthy.** Restart Patroni only on a server that
  you know is not a running primary: `sudo systemctl restart patroni`. On a
  primary, a stop can make the watchdog reset the server. If you cannot find
  the role, or there is still no leader, contact support.
- **A replica stays in `stopped`, `crashed` or `start failed`.** Find the
  cause first. If the replica needs a rebuild, recover each other member
  that is down first. Then follow
  [Rebuild a member](/docs/v2026.11.5-beta/operations/node-power-loss#rebuild-a-member). A rebuild
  deletes the data directory of the member, including data that is not on
  the leader.

Then continue with step 5 of [Recovery](#recovery).

## Prevention

- Put the three etcd members and the PostgreSQL servers in different failure
  domains: hosts, racks or power feeds. etcd loses quorum when two members
  stop.
- Give the servers a UPS or a clean shutdown path.
- Keep [backups](/docs/v2026.11.5-beta/backups) current. If the outage also damages the disks,
  you must restore from a backup.
