Operations
Recovery runbooks for HA services and the console, with symptoms and recovery times, plus maintenance, backup and restore procedures.
The recovery times come from a three-node HA service with the default Patroni
settings (ttl 30 s, loop_wait 10 s). Each time starts at the fault.
| Fault | What you see | Measured recovery | Runbook |
|---|---|---|---|
| Disk full on the primary | Writes fail with disk_full. PostgreSQL stops and Patroni fails over. |
40 s. The old primary rejoined as a replica. | Disk full on the primary |
| Disk full on a replica | The replica shows start failed. The primary keeps serving. |
47 s after space was freed. | Disk full on a replica |
| etcd (DCSDCSDistributed configuration store. etcd keeps the leader of each HA service.External DCS) disk full | One member full: no visible effect. Two members full: quorum is lost. | With failsafe mode on (the default), the primary keeps serving while every member answers it. After you free space, the member rejoins. | etcd disk full |
| Server power loss | Connections through the VIPVIPVirtual IP address. keepalived keeps it on the current primary.Virtual IP drop once, then reach the new primary. | Primary: failover 45 s, VIP 50 s, full recovery 92 s, 0 acknowledged rows lost. Replica: 75 s, 0 client errors. | Node power loss |
| Network partition | The isolated server stops PostgreSQL. The VIP moves to a surviving server. | Primary: failover 38 s, VIP 38 s, 0 acknowledged rows lost. | Network partition |
| Site outage | Nothing answers until the servers are back. Recovery order: etcd, leader, replicas, VIP. | All servers at once: full health 66 s. Staggered start (Advanced HA, one PostgreSQL server up 3 minutes before the DCS): 287 s. | Site outage and recovery |
| HAProxy or PgBouncer failure | Database service health shows HAProxy needs attention or PgBouncer needs attention. An HAProxy stop moves the VIP. | PgBouncer: 1 s after the restart. HAProxy: VIP moved in 12 s, recovery 18 s. | HAProxy or PgBouncer failure |
| Console outage | The UI and API are unreachable, or each change fails with a 500 error. Client traffic, replication and failover continue over direct mesh connections. | Crash: restart in about 6 s. Stopped console: no client errors, API back 52 s after the start. Full disk: back 8 s after space was freed. Read-only database: back 17 s after the file was writable. | Console outage |
The disk-full runbooks free space with the balloon files first.
Other runbooks:
- Replace a server: move a member slot of an HA service or a DCS cluster to another server.
- Resize a server in place: add CPU or RAM with a rolling power-off.
- Change IP addresses: a new LAN address for a server or a new address for the console.
- Upgrade a service: new versions of PostgreSQL, the extensions, Patroni and PgBouncer, from the Updates tab.
- Upgrade the PostgreSQL major: a newer PostgreSQL major in a maintenance window, from the Updates tab.
- Backups found corrupt pages: find the corrupt files and get a clean copy of the data.
- Restore the console: restore a console backup in place or on a new host.