Skip to content
You're viewing documentation for v2026.11.8-beta. It may be out of date.Go to latest
TimescaleDB Enterprise
Esc
↑↓navigate↵open⌘Jpreview
On this page

Operations

Recovery runbooks for HA services and the console, with symptoms and recovery times, plus maintenance, backup and restore procedures.

The recovery times come from a three-node HA service with the default Patroni settings (ttl 30 s, loop_wait 10 s). Each time starts at the fault.

Fault What you see Measured recovery Runbook
Disk full on the primary Writes fail with disk_full. PostgreSQL stops and Patroni fails over. 40 s. The old primary rejoined as a replica. Disk full on the primary
Disk full on a replica The replica shows start failed. The primary keeps serving. 47 s after space was freed. Disk full on a replica
etcd (DCSDCSDistributed configuration store. etcd keeps the leader of each HA service.External DCS) disk full One member full: no visible effect. Two members full: quorum is lost. With failsafe mode on (the default), the primary keeps serving while every member answers it. After you free space, the member rejoins. etcd disk full
Server power loss Connections through the VIPVIPVirtual IP address. keepalived keeps it on the current primary.Virtual IP drop once, then reach the new primary. Primary: failover 45 s, VIP 50 s, full recovery 92 s, 0 acknowledged rows lost. Replica: 75 s, 0 client errors. Node power loss
Network partition The isolated server stops PostgreSQL. The VIP moves to a surviving server. Primary: failover 38 s, VIP 38 s, 0 acknowledged rows lost. Network partition
Site outage Nothing answers until the servers are back. Recovery order: etcd, leader, replicas, VIP. All servers at once: full health 66 s. Staggered start (Advanced HA, one PostgreSQL server up 3 minutes before the DCS): 287 s. Site outage and recovery
HAProxy or PgBouncer failure Database service health shows HAProxy needs attention or PgBouncer needs attention. An HAProxy stop moves the VIP. PgBouncer: 1 s after the restart. HAProxy: VIP moved in 12 s, recovery 18 s. HAProxy or PgBouncer failure
Console outage The UI and API are unreachable, or each change fails with a 500 error. Client traffic, replication and failover continue over direct mesh connections. Crash: restart in about 6 s. Stopped console: no client errors, API back 52 s after the start. Full disk: back 8 s after space was freed. Read-only database: back 17 s after the file was writable. Console outage

The disk-full runbooks free space with the balloon files first.

Other runbooks: