Skip to content
You're viewing documentation for v2026.11.6-beta. It may be out of date.Go to latest
TimescaleDB Enterprise
Esc
↑↓navigate↵open⌘Jpreview
On this page

etcd disk full

Recover an etcd (DCS) member whose data disk is full: free space without deleting etcd data, let the member rejoin, and confirm quorum.

What happens

etcd writes to /var/lib/etcd. When that disk is full, etcd on the server stops. This can occur some time after the disk becomes full.

  • One member full. The two other members keep quorum. Clients see no error.
  • Two members full. etcd has no quorum. The primary continues to serve while each other member still sees it as the leader. If a member does not answer, the primary demotes, and writes stop until quorum is back.

On a Standard HA service, etcd runs on the PostgreSQL nodes. If /var/lib/etcd and the PostgreSQL data are on the same disk, PostgreSQL is also out of space. See Disk full on a replica and Disk full on the primary.

Symptoms

  • On the service Overview, Database service health shows etcd needs attention, and the topology shows etcd as unhealthy on the node.
  • On Advanced HA, the cluster on Infrastructure → DCS clusters shows Degraded (one member down) or Quorum lost.
  • On the server, df -h /var/lib/etcd shows the filesystem at 100%.

Recovery

Stop the applications

If two members are full, or you use Free emergency space, stop the applications that write.

Free space on the etcd disk

Free space on the disk that holds /var/lib/etcd.

Warning: Do not delete files in /var/lib/etcd. Delete only files that you know are safe to delete. If you cannot free space safely, contact support.

On a PostgreSQL node, a balloon file frees space only if it is on the same disk: compare df /var/lib/etcd with df on the directory of the balloon file. A DCS server has no balloon files. Without quorum, Free emergency space can fail to set the read-only default. Do not wait for it.

Wait for etcd

Wait for etcd to start again. systemd tries every 10 seconds. The member then rejoins the cluster.

Confirm that etcd is healthy

Confirm that etcd is healthy in the service topology, or on the DCS cluster page. If the console cannot reach etcd, use step 3 of Site outage and recovery. On a single-node service, etcd uses TLS: use https://, and add --cacert=/etc/etcd/tls/ca.crt --cert=/etc/etcd/tls/server.crt --key=/etc/etcd/tls/server.key.

Check quorum

If etcd does not start after you free space, check quorum first.

Check the service health

Confirm that Database service health is healthy. If the console cannot show it, use step 4 of Site outage and recovery. If there is no leader, follow Patroni does not elect a leader or a replica does not rejoin.

Allow writes again

If you used a balloon file or Free emergency space, do Return to service, and recreate the balloon. If the read-only default was not set, first set it by hand as in Before you free space.

Start the applications

Start the applications that you stopped.

Prevention

  • Put /var/lib/etcd on its own disk, or set an alert for free space below 20% on its filesystem.
  • Keep the balloon files on each PostgreSQL node.