Console outage
Recover a stopped console, a full console disk, or a console database that is read-only or damaged, and see what keeps working meanwhile.
What keeps working
The console is not on the data path. While it is down, these continue on the nodes:
- Client traffic through the VIP and HAProxy (port 5000 pooled, port 6000 direct). See Connect.
- Replication and failover. See High availability.
- Mesh connections between nodes, while no node restarts.
- Scheduled backups and WAL archiving, while the backup server does not restart.
What stops
The console UI and the API, all operations (deploys, configuration changes, manual backups, restores, forks, server and LiveSync changes), server enrollment, metrics collection (a gap in Monitoring) and alerts.
A node restarts during the outage
A node that restarts (reboot, power loss, or a restart of tse-tailscaled)
cannot join the mesh again until the console is back. PostgreSQL, Patroni
and etcd on that node cannot reach the cluster.
| Nodes that restart | Effect |
|---|---|
| One node (Standard HA) | The service continues on the two other nodes. |
| Two nodes (Standard HA) | etcd loses quorum. The leader can demote, and writes stop. No new leader until quorum returns. |
| Leader, while a replica cannot reach the cluster (Advanced HA) | No PostgreSQL node is healthy. The service stops. |
To confirm, run this on the restarted node:
sudo /opt/tse/tailscale/bin/tailscale --socket=/opt/tse/tailscale/tailscaled.sock status
Bad: the output shows You are logged out, or fetch control key in the
last error.
Do not restart a node. Recover the console (see Recovery). The nodes then join the mesh, and the cluster recovers in about one minute.
A backup server restarts during the outage
A backup server that restarts cannot join the mesh again until the console is back. This also applies to the node of a single-node service.
- WAL archiving stops for each service that uses the backup server. The
primary keeps the WAL in
pg_wal. If the primary disk becomes full, writes stop. See Disk full on the primary. - Scheduled backups to the backup server fail. A failed backup does not run again until its next scheduled time.
To confirm, run this on the running primary:
sudo -u postgres psql -Xc "SELECT failed_count, last_failed_time, last_archived_time FROM pg_stat_archiver"
- Bad:
failed_countincreases, andlast_failed_timeis recent. - Good (after the recovery):
last_archived_timeadvances.
Recover the console (see Recovery), and run the query again. If a scheduled backup failed, start a manual backup (see Backups).
Symptoms
Run the checks on the console host. The commands use
http://localhost:8080. If TLS is on (TSE_TLS_MODE is self-signed or
provided), use curl -sk https://localhost/....
curl -s http://localhost:8080/api/v1/infofails.docker compose -f docker-compose.yml psshows theappservice asexited,restartingor missing.- Disk full: reads work, but each change fails with a 500 error:
database or disk is fullordisk I/O error. Server registration can fail with502 could not mint enroll key.df -hshows the filesystem of the Docker volumes at 100%. - Read-only database: each change fails with a 500 error
(
attempt to write a readonly database). After a restart, the console runs in degraded mode:/api/v1/infoanswers 200 with adegraded.databasefield, and every other API call, login included, answers 503. - Damaged database: each change fails with
database disk image is malformed.
Recovery
Recover the console first. Do not restart nodes to repair the database service.
If the console host is lost, go to step 6. Otherwise, run the commands on the
console host, in the install directory (default /opt/tse). Do step 1, then
the one step of 2 to 5 that matches the problem.
Read the container state
Read the container state and its log:
docker compose -f docker-compose.yml ps
docker compose -f docker-compose.yml logs --tail=200 appStart the container
The container does not run. Start it, and read the API status:
docker compose -f docker-compose.yml up -d app
curl -si http://localhost:8080/api/v1/infoGood: 200. If the body has a degraded.database field, go to step 4.
Otherwise, log in, and confirm that the services show running and
healthy.
Free space on the console disk
The console disk is full. Free space with your own retention or disk expansion procedure.
Warning: Do not delete console database files or backup files to make space.
Check the free space, and restart the app:
df -h /var/lib/docker
docker compose -f docker-compose.yml restart appTo prove that writes work, open Administration → Console backups, select Back up now, and confirm that the backup completes.
Repair the read-only database
The database is read-only. Find the volume that holds /app/data
(see Data persistence), and check it:
vol=$(docker volume inspect -f '{{.Mountpoint}}' "$(docker compose -f docker-compose.yml ps -q app | xargs docker inspect -f '{{range .Mounts}}{{if eq .Destination "/app/data"}}{{.Name}}{{end}}{{end}}')")
lsattr "$vol"/console.db*
mount | grep "$(df --output=source "$vol" | tail -1)"
dmesg | grep -i -E 'ext4|xfs|I/O error' | tailiin thelsattroutput (immutable flag): remove the flag from that file withchattr -i <file>.roin themountoutput (read-only filesystem): repair the disk, then mount the filesystem read-write again.
After the repair, the console opens the database within 5 seconds, with
no restart. The degraded field goes from /api/v1/info, and login
works. Do not restore or delete the database only because this check
failed.
Restore the damaged database
The database is damaged.
Warning: Do not delete the database file. It holds the servers, services, backup configuration and mesh state.
In the install directory, run ./timescale-enterprise.sh, and choose
Restore the console from a backup. Enter the newest console backup and its recovery key
file. See
Restore on a new host
for the prerequisites and the checks. The installer first backs up the
damaged console to <TSE_BACKUP_DIR>/pre-restore/. If that backup fails,
it stops and changes nothing.
Only if that pre-restore backup fails, use this fallback.
Warning: The fallback deletes the damaged console volumes and .env,
with no backup of them. First confirm that the backup file and its
recovery key are outside those volumes. If you cannot confirm this, or you
need data from the damaged console, stop and contact support.
- Note the backup directory:
grep '^TSE_BACKUP_DIR=' .env. Put the backup file in it. - Run the installer, choose Uninstall, and type
uninstall. The uninstall keeps the backup directory. - Run the installer again, and choose Restore the console from a backup.
- At the
Directoryprompt, enter the backup directory that you noted. If not, the console does not find its local backups. - Do the checks of Restore on a new host.
The options for the same steps are --uninstall and
--restore <file>. With no terminal, TSE_INSTALL_BACKUP_DIR sets the
backup directory.
If you have no console backup, open a support case.
Restore on a standby host
The console host is lost. Restore the newest console backup on a standby host with Restore the console. Keep the old console stopped.
Prevention
- Monitor free space on the filesystem of the Docker volumes. Metrics history (Monitoring settings), logs and support reports use this space.
- Set up console backups to an S3 target, with the daily schedule and Back up after changes on. Keep the recovery key away from the console host.
- Put the console on a different host and power supply from the nodes.
- Monitor
/api/v1/infofrom outside the console: a stopped console sends no alerts. - Do not reboot nodes or restart
tse-tailscaledwhile the console is down. - The console starts again after a host reboot. A console that you stopped stays stopped.