---
title: Operations
description: "Recovery runbooks for HA services and the console, with symptoms and recovery times, plus maintenance, backup and restore procedures."
---

The recovery times come from a three-node HA service with the default Patroni
settings (`ttl` 30 s, `loop_wait` 10 s). Each time starts at the fault.

| Fault | What you see | Measured recovery | Runbook |
| --- | --- | --- | --- |
| Disk full on the primary | Writes fail with `disk_full`. PostgreSQL stops and Patroni fails over. | 40 s. The old primary rejoined as a replica. | [Disk full on the primary](/docs/v2026.11.8-beta/operations/disk-full-primary) |
| Disk full on a replica | The replica shows `start failed`. The primary keeps serving. | 47 s after space was freed. | [Disk full on a replica](/docs/v2026.11.8-beta/operations/disk-full-replica) |
| etcd (DCS (DCS: Distributed configuration store. etcd keeps the leader of each HA service.)) disk full | One member full: no visible effect. Two members full: quorum is lost. | With failsafe mode on (the default), the primary keeps serving while every member answers it. After you free space, the member rejoins. | [etcd disk full](/docs/v2026.11.8-beta/operations/dcs-disk-full) |
| Server power loss | Connections through the VIP (VIP: Virtual IP address. keepalived keeps it on the current primary.) drop once, then reach the new primary. | Primary: failover 45 s, VIP 50 s, full recovery 92 s, 0 acknowledged rows lost. Replica: 75 s, 0 client errors. | [Node power loss](/docs/v2026.11.8-beta/operations/node-power-loss) |
| Network partition | The isolated server stops PostgreSQL. The VIP moves to a surviving server. | Primary: failover 38 s, VIP 38 s, 0 acknowledged rows lost. | [Network partition](/docs/v2026.11.8-beta/operations/network-partition) |
| Site outage | Nothing answers until the servers are back. Recovery order: etcd, leader, replicas, VIP. | All servers at once: full health 66 s. Staggered start (Advanced HA, one PostgreSQL server up 3 minutes before the DCS): 287 s. | [Site outage and recovery](/docs/v2026.11.8-beta/operations/site-outage) |
| HAProxy or PgBouncer failure | **Database service health** shows *HAProxy needs attention* or *PgBouncer needs attention*. An HAProxy stop moves the VIP. | PgBouncer: 1 s after the restart. HAProxy: VIP moved in 12 s, recovery 18 s. | [HAProxy or PgBouncer failure](/docs/v2026.11.8-beta/operations/proxy-failure) |
| Console outage | The UI and API are unreachable, or each change fails with a 500 error. Client traffic, replication and failover continue over direct mesh connections. | Crash: restart in about 6 s. Stopped console: no client errors, API back 52 s after the start. Full disk: back 8 s after space was freed. Read-only database: back 17 s after the file was writable. | [Console outage](/docs/v2026.11.8-beta/operations/console-outage) |

The disk-full runbooks free space with the [balloon files](/docs/v2026.11.8-beta/operations/balloon-file) first.

Other runbooks:

- [Replace a server](/docs/v2026.11.8-beta/operations/replace-a-server): move a member slot of an
  HA service or a DCS cluster to another server.
- [Resize a server in place](/docs/v2026.11.8-beta/operations/resize-in-place): add CPU or RAM
  with a rolling power-off.
- [Change IP addresses](/docs/v2026.11.8-beta/operations/change-ip-addresses): a new LAN address
  for a server or a new address for the console.
- [Upgrade a service](/docs/v2026.11.8-beta/operations/upgrade-a-service): new versions of
  PostgreSQL, the extensions, Patroni and PgBouncer, from the **Updates** tab.
- [Upgrade the PostgreSQL major](/docs/v2026.11.8-beta/operations/upgrade-the-postgresql-major):
  a newer PostgreSQL major in a maintenance window, from the **Updates** tab.
- [Backups found corrupt pages](/docs/v2026.11.8-beta/operations/backups-found-corrupt-pages):
  find the corrupt files and get a clean copy of the data.
- [Restore the console](/docs/v2026.11.8-beta/operations/restore-the-console): restore a console
  backup in place or on a new host.
