Skip to content
You're viewing documentation for v2026.11.8-beta. It may be out of date.Go to latest
TimescaleDB Enterprise
Esc
↑↓navigate↵open⌘Jpreview
On this page

Site outage and recovery

Bring back an HA service after all of its servers lost power at the same time, confirm that it has one primary, and check for lost commits.

What happens

All servers of the HA service stop at the same time, for example after a power loss or a storage failure. When they start again, the service recovers by itself. A leader is back in about 40 s, and the service is healthy in about one minute.

A server that starts alone does not promote. On an Advanced HA service, PostgreSQL stays stopped until the DCS servers have quorum.

Symptoms

  • Clients cannot connect through the VIP or to any server. Connections are refused or time out.
  • On the service Overview, Database service health shows etcd, Patroni, and HAProxy need attention.

Recovery

Recover the console

If the console is down, recover it first with Console outage. The servers cannot join the mesh until the console is available.

Power on all servers

Power on all servers. On Advanced HA, start the DCS servers first. If a server cannot start, do the next checks on the servers that run.

Confirm etcd quorum

In the topology on the service Overview, etcd must be healthy on at least two nodes. On Advanced HA, the cluster on Infrastructure → DCS clusters must not show Quorum lost.

If the console cannot show it, run this on a running DCS member (on Standard HA, a PostgreSQL server):

sudo /usr/local/bin/etcdctl --endpoints=http://127.0.0.1:2379 endpoint health --cluster
  • Good: at least two members show is healthy.
  • Bad: fewer than two, or the command fails. Go to etcd has no quorum.

Confirm the Patroni cluster

In the topology, one member must be the leader, and each other member a replica with a lag near 0.

If the console cannot show it, run this on a running PostgreSQL server:

sudo -u postgres patronictl -c /etc/patroni/patroni.yml list

Check the service health

Confirm that Database service health is healthy.

Connect from a client

From a client, connect to port 5000 of the VIP (or of the node endpoints without a VIP), and confirm that you reach the leader. See Connect. If the VIP does not answer, run sudo systemctl status keepalived haproxy on each PostgreSQL server, and read the log of a failed unit.

Replace a lost server

If a server is lost for good, use Replace a server.

If a step does not recover, contact support.

If it does not heal

etcd has no quorum

Patroni cannot elect a leader while fewer than 2 of 3 etcd members are healthy.

Start the etcd members

Start each etcd member that is down. Do not remove or replace members by hand.

Read the etcd log

If a member does not start, read its log on that server: sudo journalctl -u etcd --since "1 hour ago".

Continue the recovery

When two members are healthy, continue with step 4 of Recovery.

If a member cannot come back:

  • One member is lost with its server, and the other two are healthy: use Replace a server.
  • Two members are lost, or a member does not start for another reason: create a support report and open a support case.

Patroni does not elect a leader or a replica does not rejoin

Read the Patroni log of the server: on Logs & Diagnostics, select the server and the log source Patroni. If the console cannot show it, run sudo tail -n 200 /var/log/patroni/patroni.log on the server.

  • PostgreSQL is stopped and the log shows etcd errors. Restore etcd quorum first. Patroni starts PostgreSQL when etcd answers.
  • No leader while etcd is healthy. Restart Patroni only on a server that you know is not a running primary: sudo systemctl restart patroni. On a primary, a stop can make the watchdog reset the server. If you cannot find the role, or there is still no leader, contact support.
  • A replica stays in stopped, crashed or start failed. Find the cause first. If the replica needs a rebuild, recover each other member that is down first. Then follow Rebuild a member. A rebuild deletes the data directory of the member, including data that is not on the leader.

Then continue with step 5 of Recovery.

Prevention

  • Put the three etcd members and the PostgreSQL servers in different failure domains: hosts, racks or power feeds. etcd loses quorum when two members stop.
  • Give the servers a UPS or a clean shutdown path.
  • Keep backups current. If the outage also damages the disks, you must restore from a backup.