Skip to content
TimescaleDB Enterprise
Esc
↑↓navigate↵open⌘Jpreview
On this page

Console outage

Recover a stopped console, a full console disk, or a console database that is read-only or damaged, and see what keeps working meanwhile.

What keeps working

The console is not on the data path. While it is down, these continue on the nodes:

  • Client traffic through the VIP and HAProxy (port 5000 pooled, port 6000 direct). See Connect.
  • Replication and failover. See High availability.
  • Mesh connections between nodes, while no node restarts.
  • Scheduled backups and WAL archiving, while the backup server does not restart.

What stops

The console UI and the API, all operations (deploys, configuration changes, manual backups, restores, forks, server and LiveSync changes), server enrollment, metrics collection (a gap in Monitoring) and alerts.

A node restarts during the outage

A node that restarts (reboot, power loss, or a restart of tse-tailscaled) cannot join the mesh again until the console is back. PostgreSQL, Patroni and etcd on that node cannot reach the cluster.

Nodes that restart Effect
One node (Standard HA) The service continues on the two other nodes.
Two nodes (Standard HA) etcd loses quorum. The leader can demote, and writes stop. No new leader until quorum returns.
Leader, while a replica cannot reach the cluster (Advanced HA) No PostgreSQL node is healthy. The service stops.

To confirm, run this on the restarted node:

sudo /opt/tse/tailscale/bin/tailscale --socket=/opt/tse/tailscale/tailscaled.sock status

Bad: the output shows You are logged out, or fetch control key in the last error.

Do not restart a node. Recover the console (see Recovery). The nodes then join the mesh, and the cluster recovers in about one minute.

A backup server restarts during the outage

A backup server that restarts cannot join the mesh again until the console is back. This also applies to the server of a Standard service.

  • WAL archiving stops for each service that uses the backup server. The primary keeps the WAL in pg_wal. If the primary disk becomes full, writes stop. See Disk full on the primary.
  • Scheduled backups to the backup server fail. A failed backup does not run again until its next scheduled time.

To confirm, run this on the running primary:

sudo -u postgres psql -Xc "SELECT failed_count, last_failed_time, last_archived_time FROM pg_stat_archiver"
  • Bad: failed_count increases, and last_failed_time is recent.
  • Good (after the recovery): last_archived_time advances.

Recover the console (see Recovery), and run the query again. If a scheduled backup failed, start a manual backup (see Backups).

Symptoms

Run the checks on the console host. The commands use http://localhost:8080. If TLS is on (TSE_TLS_MODE is self-signed or provided), use curl -sk https://localhost/....

  • curl -s http://localhost:8080/api/v1/info fails.
  • docker compose -f docker-compose.yml ps shows the app service as exited, restarting or missing.
  • Disk full: reads work, but each change fails with a 500 error: database or disk is full or disk I/O error. Server registration can fail with 502 could not mint enroll key. df -h shows the filesystem of the Docker volumes at 100%.
  • Read-only database: each change fails with a 500 error (attempt to write a readonly database). After a restart, the console runs in degraded mode: /api/v1/info answers 200 with a degraded.database field, and every other API call, login included, answers 503.
  • Damaged database: each change fails with database disk image is malformed.

Recovery

Recover the console first. Do not restart nodes to repair the database service.

If the console host is lost, go to step 6. Otherwise, run the commands on the console host, in the install directory (default /opt/tse). Do step 1, then the one step of 2 to 5 that matches the problem.

Read the container state

Read the container state and its log:

docker compose -f docker-compose.yml ps
docker compose -f docker-compose.yml logs --tail=200 app

Start the container

The container does not run. Start it, and read the API status:

docker compose -f docker-compose.yml up -d app
curl -si http://localhost:8080/api/v1/info

Good: 200. If the body has a degraded.database field, go to step 4. Otherwise, log in, and confirm that the services show running and healthy.

Free space on the console disk

The console disk is full. Free space with your own retention or disk expansion procedure.

Warning: Do not delete console database files or backup files to make space.

Check the free space, and restart the app:

df -h /var/lib/docker
docker compose -f docker-compose.yml restart app

To prove that writes work, open Administration → Console backups, select Back up now, and confirm that the backup completes.

Repair the read-only database

The database is read-only. Find the volume that holds /app/data (see Data persistence), and check it:

vol=$(docker volume inspect -f '{{.Mountpoint}}' "$(docker compose -f docker-compose.yml ps -q app | xargs docker inspect -f '{{range .Mounts}}{{if eq .Destination "/app/data"}}{{.Name}}{{end}}{{end}}')")
lsattr "$vol"/console.db*
mount | grep "$(df --output=source "$vol" | tail -1)"
dmesg | grep -i -E 'ext4|xfs|I/O error' | tail
  • i in the lsattr output (immutable flag): remove the flag from that file with chattr -i <file>.
  • ro in the mount output (read-only filesystem): repair the disk, then mount the filesystem read-write again.

After the repair, the console opens the database within 5 seconds, with no restart. The degraded field goes from /api/v1/info, and login works. Do not restore or delete the database only because this check failed.

Restore the damaged database

The database is damaged.

Warning: Do not delete the database file. It holds the servers, services, backup configuration and mesh state.

In the install directory, run ./timescale-enterprise.sh, and choose Restore the console from a backup. Enter the newest console backup and its recovery key file. See Restore on a new host for the prerequisites and the checks. The installer first backs up the damaged console to <TSE_BACKUP_DIR>/pre-restore/. If that backup fails, it stops and changes nothing.

Only if that pre-restore backup fails, use this fallback.

Warning: The fallback deletes the damaged console volumes and .env, with no backup of them. First confirm that the backup file and its recovery key are outside those volumes. If you cannot confirm this, or you need data from the damaged console, stop and contact support.

  1. Note the backup directory: grep '^TSE_BACKUP_DIR=' .env. Put the backup file in it.
  2. Run the installer, choose Uninstall, and type uninstall. The uninstall keeps the backup directory.
  3. Run the installer again, and choose Restore the console from a backup.
  4. At the Directory prompt, enter the backup directory that you noted. If not, the console does not find its local backups.
  5. Do the checks of Restore on a new host.

The options for the same steps are --uninstall and --restore <file>. With no terminal, TSE_INSTALL_BACKUP_DIR sets the backup directory.

If you have no console backup, open a support case.

Restore on a standby host

The console host is lost. Restore the newest console backup on a standby host with Restore the console. Keep the old console stopped.

Prevention

  • Monitor free space on the filesystem of the Docker volumes. Metrics history (Monitoring settings), logs and support reports use this space.
  • Set up console backups to an S3 target, with the daily schedule and Back up after changes on. Keep the recovery key away from the console host.
  • Put the console on a different host and power supply from the nodes.
  • Monitor /api/v1/info from outside the console: a stopped console sends no alerts.
  • Do not reboot nodes or restart tse-tailscaled while the console is down.
  • The console starts again after a host reboot. A console that you stopped stays stopped.