etcd disk full
Recover an etcd (DCS) member whose data disk is full: free space without deleting etcd data, let the member rejoin, and confirm quorum.
What happens
etcd writes to /var/lib/etcd. When that disk is full, etcd on the server
stops. This can occur some time after the disk becomes full.
- One member full. The two other members keep quorum. Clients see no error.
- Two members full. etcd has no quorum. The primary continues to serve while each other member still sees it as the leader. If a member does not answer, the primary demotes, and writes stop until quorum is back.
On a Standard HA service, etcd runs on the PostgreSQL nodes. If
/var/lib/etcd and the PostgreSQL data are on the same disk, PostgreSQL is
also out of space. See Disk full on a replica
and Disk full on the primary.
Symptoms
- On the service Overview, Database service health shows etcd needs attention, and the topology shows etcd as unhealthy on the node.
- On Advanced HA, the cluster on Infrastructure → DCS clusters shows Degraded (one member down) or Quorum lost.
- On the server,
df -h /var/lib/etcdshows the filesystem at 100%.
Recovery
Stop the applications
If two members are full, or you use Free emergency space, stop the applications that write.
Free space on the etcd disk
Free space on the disk that holds /var/lib/etcd.
Warning: Do not delete files in /var/lib/etcd. Delete only files
that you know are safe to delete. If you cannot free space safely,
contact support.
On a PostgreSQL node, a balloon file frees
space only if it is on the same disk: compare df /var/lib/etcd with df
on the directory of the balloon file. A DCS server has no balloon files.
Without quorum, Free emergency space can fail to set the read-only
default. Do not wait for it.
Wait for etcd
Wait for etcd to start again. systemd tries every 10 seconds. The member then rejoins the cluster.
Confirm that etcd is healthy
Confirm that etcd is healthy in the service topology, or on the DCS
cluster page. If the console cannot reach etcd, use step 3 of
Site outage and recovery. On a
Standard service, etcd uses TLS: use https://, and add
--cacert=/etc/etcd/tls/ca.crt --cert=/etc/etcd/tls/server.crt --key=/etc/etcd/tls/server.key.
Check quorum
If etcd does not start after you free space, check quorum first.
- Two members are healthy: replace the member with Replace a server (for a DCS cluster, DCS member).
- No quorum: do not replace members by hand. Follow etcd has no quorum.
Check the service health
Confirm that Database service health is healthy. If the console cannot show it, use step 4 of Site outage and recovery. If there is no leader, follow Patroni does not elect a leader or a replica does not rejoin.
Allow writes again
If you used a balloon file or Free emergency space, do Return to service, and recreate the balloon. If the read-only default was not set, first set it by hand as in Before you free space.
Start the applications
Start the applications that you stopped.
Prevention
- Put
/var/lib/etcdon its own disk, or set an alert for free space below 20% on its filesystem. - Keep the balloon files on each PostgreSQL node.