---
title: Console outage
description: "Recover a stopped console, a full console disk, or a console database that is read-only or damaged, and see what keeps working meanwhile."
related:
  - /operations/restore-the-console
  - /console-backups
---

## What keeps working

The console is not on the data path. While it is down, these continue on the
nodes:

- Client traffic through the VIP and HAProxy (port 5000 pooled, port 6000
  direct). See [Connect](/docs/v2026.11.13-beta/database-services/connect).
- Replication and failover. See
  [High availability](/docs/v2026.11.13-beta/database-services/high-availability).
- Mesh connections between nodes, while no node restarts.
- Scheduled backups and WAL archiving, while the backup server does not
  restart.

## What stops

The console UI and the API, all operations (deploys, configuration changes,
manual backups, restores, forks, server and LiveSync changes), server
enrollment, metrics collection (a gap in [Monitoring](/docs/v2026.11.13-beta/monitoring)) and
alerts.

## A node restarts during the outage

A node that restarts (reboot, power loss, or a restart of `tse-tailscaled`)
cannot join the mesh again until the console is back. PostgreSQL, Patroni
and etcd on that node cannot reach the cluster.

| Nodes that restart | Effect |
| --- | --- |
| One node (Standard HA) | The service continues on the two other nodes. |
| Two nodes (Standard HA) | etcd loses quorum. The leader can demote, and writes stop. No new leader until quorum returns. |
| Leader, while a replica cannot reach the cluster (Advanced HA) | No PostgreSQL node is healthy. The service stops. |

To confirm, run this on the restarted node:

```bash
sudo /opt/tse/tailscale/bin/tailscale --socket=/opt/tse/tailscale/tailscaled.sock status
```

Bad: the output shows `You are logged out`, or `fetch control key` in the
last error.

Do not restart a node. Recover the console (see [Recovery](#recovery)). The
nodes then join the mesh, and the cluster recovers in about one minute.

## A backup server restarts during the outage

A backup server that restarts cannot join the mesh again until the console is
back. This also applies to the server of a Standard service.

- WAL archiving stops for each service that uses the backup server. The
  primary keeps the WAL in `pg_wal`. If the primary disk becomes full, writes
  stop. See [Disk full on the primary](/docs/v2026.11.13-beta/operations/disk-full-primary).
- Scheduled backups to the backup server fail. A failed backup does not run
  again until its next scheduled time.

To confirm, run this on the running primary:

```bash
sudo -u postgres psql -Xc "SELECT failed_count, last_failed_time, last_archived_time FROM pg_stat_archiver"
```

- Bad: `failed_count` increases, and `last_failed_time` is recent.
- Good (after the recovery): `last_archived_time` advances.

Recover the console (see [Recovery](#recovery)), and run the query again. If
a scheduled backup failed, start a manual backup (see [Backups](/docs/v2026.11.13-beta/backups)).

## Symptoms

Run the checks on the console host. The commands use
`http://localhost:8080`. If TLS is on (`TSE_TLS_MODE` is `self-signed` or
`provided`), use `curl -sk https://localhost/...`.

- `curl -s http://localhost:8080/api/v1/info` fails.
- `docker compose -f docker-compose.yml ps` shows the `app` service as
  `exited`, `restarting` or missing.
- **Disk full:** reads work, but each change fails with a 500 error:
  `database or disk is full` or `disk I/O error`. Server registration can
  fail with `502 could not mint enroll key`. `df -h` shows the filesystem of
  the Docker volumes at 100%.
- **Read-only database:** each change fails with a 500 error
  (`attempt to write a readonly database`). After a restart, the console runs
  in degraded mode: `/api/v1/info` answers 200 with a `degraded.database`
  field, and every other API call, login included, answers 503.
- **Damaged database:** each change fails with
  `database disk image is malformed`.

## Recovery

Recover the console first. Do not restart nodes to repair the database
service.

If the console host is lost, go to step 6. Otherwise, run the commands on the
console host, in the install directory (default `/opt/tse`). Do step 1, then
the one step of 2 to 5 that matches the problem.

1. **Read the container state**

    Read the container state and its log:

    ```bash
    docker compose -f docker-compose.yml ps
    docker compose -f docker-compose.yml logs --tail=200 app
    ```

2. **Start the container**

    **The container does not run.** Start it, and read the API status:

    ```bash
    docker compose -f docker-compose.yml up -d app
    curl -si http://localhost:8080/api/v1/info
    ```

    Good: `200`. If the body has a `degraded.database` field, go to step 4.
    Otherwise, log in, and confirm that the services show `running` and
    healthy.

3. **Free space on the console disk**

    **The console disk is full.** Free space with your own retention or disk
    expansion procedure.

    **Warning:** Do not delete console database files or backup files to make
    space.

    Check the free space, and restart the app:

    ```bash
    df -h /var/lib/docker
    docker compose -f docker-compose.yml restart app
    ```

    To prove that writes work, open **Administration → Console backups**,
    select **Back up now**, and confirm that the backup completes.

4. **Repair the read-only database**

    **The database is read-only.** Find the volume that holds `/app/data`
    (see [Data persistence](/docs/v2026.11.13-beta/configuration#data-persistence)), and check it:

    ```bash
    vol=$(docker volume inspect -f '{{.Mountpoint}}' "$(docker compose -f docker-compose.yml ps -q app | xargs docker inspect -f '{{range .Mounts}}{{if eq .Destination "/app/data"}}{{.Name}}{{end}}{{end}}')")
    lsattr "$vol"/console.db*
    mount | grep "$(df --output=source "$vol" | tail -1)"
    dmesg | grep -i -E 'ext4|xfs|I/O error' | tail
    ```

    - `i` in the `lsattr` output (immutable flag): remove the flag from that
      file with `chattr -i <file>`.
    - `ro` in the `mount` output (read-only filesystem): repair the disk, then
      mount the filesystem read-write again.

    After the repair, the console opens the database within 5 seconds, with
    no restart. The `degraded` field goes from `/api/v1/info`, and login
    works. Do not restore or delete the database only because this check
    failed.

5. **Restore the damaged database**

    **The database is damaged.**

    **Warning:** Do not delete the database file. It holds the servers,
    services, backup configuration and mesh state.

    In the install directory, run `./timescale-enterprise.sh`, and choose
    **Restore the console from a backup**. Enter the newest [console backup](/docs/v2026.11.13-beta/console-backups) and its recovery key
    file. See
    [Restore on a new host](/docs/v2026.11.13-beta/operations/restore-the-console#restore-on-a-new-host)
    for the prerequisites and the checks. The installer first backs up the
    damaged console to `<TSE_BACKUP_DIR>/pre-restore/`. If that backup fails,
    it stops and changes nothing.

    Only if that pre-restore backup fails, use this fallback.

    **Warning:** The fallback deletes the damaged console volumes and `.env`,
    with no backup of them. First confirm that the backup file and its
    recovery key are outside those volumes. If you cannot confirm this, or you
    need data from the damaged console, stop and contact support.

    1. Note the backup directory: `grep '^TSE_BACKUP_DIR=' .env`. Put the
       backup file in it.
    2. Run the installer, choose **Uninstall**, and type `uninstall`. The
       uninstall keeps the backup directory.
    3. Run the installer again, and choose **Restore the console from a
       backup**.
    4. At the `Directory` prompt, enter the backup directory that you noted.
       If not, the console does not find its local backups.
    5. Do the checks of
       [Restore on a new host](/docs/v2026.11.13-beta/operations/restore-the-console#restore-on-a-new-host).

    The options for the same steps are `--uninstall` and
    `--restore <file>`. With no terminal, `TSE_INSTALL_BACKUP_DIR` sets the
    backup directory.

    If you have no console backup, open a support case.

6. **Restore on a standby host**

    **The console host is lost.** Restore the newest console backup on a
    standby host with [Restore the console](/docs/v2026.11.13-beta/operations/restore-the-console).
    Keep the old console stopped.

## Prevention

- Monitor free space on the filesystem of the Docker volumes. Metrics
  history ([Monitoring settings](/docs/v2026.11.13-beta/monitoring#monitoring-settings)), logs and
  support reports use this space.
- Set up [console backups](/docs/v2026.11.13-beta/console-backups) to an S3 target, with the daily
  schedule and **Back up after changes** on. Keep the recovery key away from
  the console host.
- Put the console on a different host and power supply from the nodes.
- Monitor `/api/v1/info` from outside the console: a stopped console sends
  no alerts.
- Do not reboot nodes or restart `tse-tailscaled` while the console is down.
- The console starts again after a host reboot. A console that you stopped
  stays stopped.
