Skip to content
You're viewing documentation for v2026.11.6-beta. It may be out of date.Go to latest
TimescaleDB Enterprise
Esc
↑↓navigate↵open⌘Jpreview
On this page

Disk full on a replica

Free space on an HA replica with a full data disk, confirm that it streams from the primary again, and rebuild the replica if it cannot catch up.

What happens

The data disk of a replica is full. The replica cannot write WAL, and PostgreSQL on the replica can stop. Patroni restarts it, and it stops again until there is free space.

Symptoms

  • In the topology on the service Overview, the replica shows start failed, and the Patroni health is unhealthy.
  • The primary continues to serve reads and writes. Clients that use the replica can get errors.
  • In a mild case, PostgreSQL on the replica stays up, but its lag increases.
  • pg_wal on the primary grows, because the replication slot of the replica keeps WAL.
  • The Storage panel on the Monitoring tab shows the disk of the replica full or almost full.

Triage

Find what uses the space before you choose a lasting fix. To find the disk of the data directory, see Grow storage on a server.

Cause Do this
The database needs more space Grow the storage. Then go to step 2 of the recovery. Also check the primary and each other replica.
Retained WAL, logs, temporary files or other files Do the recovery steps.
You need space now, or the cause is not known Do the recovery steps. A balloon file gives time. It does not remove the cause.

Do not delete PostgreSQL data or WAL files by hand.

Recovery

Free space on the replica

Free space on the replica. Use one of these methods:

  • Console: stop the applications that write. Then choose Free emergency space in the menu of the replica in the topology. See Free space from the console. The console sets the whole cluster, also the primary, to read-only by default. The message names the server where files were deleted. If no file was deleted on the replica, run the action again on the replica.

  • By hand, to keep the primary writable: connect to the replica over SSH and delete one balloon file. If tse-balloon-1 is missing, delete tse-balloon-2. Record which file you delete.

    sudo rm <data_dir>/../tse-balloon-1

Then remove what filled the disk (see Free space). If you cannot free space or grow the disk, contact support.

Wait for the replica

Wait for the replica to recover. In the topology, it must show a healthy status dot, no state label, and a lag near 0. If it does not recover, go to step 3 or 4.

Wait for a running rebuild

If the replica shows creating replica, Patroni is rebuilding it from the primary. Wait for the rebuild to finish. Do not start another rebuild. If the rebuild fails, contact support.

Rebuild the replica

If the replica stays in start failed, or it cannot catch up, rebuild it from the primary. The console has no button for this step. Use the API (admin role): POST /api/v1/database-services/{id}/servers/{server_id}/rebuild-replica. The console refuses the primary and needs each other server healthy. See the API reference.

The rebuild deletes the data directory of the replica and copies it again from the primary. Do not rebuild a replica that is catching up. Make sure that the replica has space for the copy. If you are not sure, contact support.

Wait until the replica shows streaming with a lag near 0. If the rebuild fails, contact support.

Allow writes and restore the reserve

  • If you used Free emergency space, do steps 7 to 9 of Free space from the console: the banner, Return to service, and a test write. Return to service starts a run that recreates the reserve.
  • If you deleted a file by hand, choose Recreate emergency reserve in the menu of the replica. See Recreate the balloon.

Check the run on the Operations tab.

Check the free space

Check the free space on the primary and on each replica. Grow each disk that needs more space.

Prevention

  • Give each replica at least the disk size of the primary.
  • Keep the balloon files in place.
  • Configure alert delivery. The built-in disk rules apply to each server.