Articles
Updates, backups and restore: what the hold is for
How the panel updates itself, what a backup contains, why the key has to live off the server, and why the panel starts on hold after a restore.
About 4 min read · Last updated
In this article
How the panel updates itself
You start an update under Administration → System → Updates or from the command line:
sudo sh scripts/install.sh updateBoth run the same flow:
- An encrypted backup is taken. If it cannot be taken, nothing changes.
- The worker is stopped; it waits at most 120 seconds for the running step to finish.
- The new version is built and started, and its health is checked.
- If the new version does not become ready, the previous version comes back automatically.
The database is not rolled back automatically. Should you need it, the restore command to run is written in the update's log, with the name of the backup.
The panel has no authority to run commands on the server directly, and it is not given the Docker socket. The API leaves a request file; a systemd unit added by the installer sees the file and runs the installer script. The script validates the request again and accepts only operations from a fixed list: checking for releases, update, backup, the automatic-update setting, the domain, server settings, monitoring, the terminal and operating-system updates. Any request not on the list is refused.
Channels
| Channel | What it follows |
|---|---|
stable | Published releases only |
beta | Pre-releases too |
edge | Every change on a branch; meant for test servers |
You can switch automatic updating on with a maintenance window:
sh scripts/install.sh auto-update on --channel stable --window 03:00-05:00Read upgrade-notes.md (Turkish) before upgrading. It lists the behaviour that changes between versions, the effect on existing data, and queries that show who will be affected.
What a backup contains
The whole of PostgreSQL, .env, the master key and the VMware service token, Caddy's certificates and ACME account, your own certificate if you uploaded one, and the Alertmanager configuration. Redis is not backed up; it holds only copies of queues, single-use tickets and rate-limit counters.
A backup does not contain customer VMs. The docs add a reminder: a VMware snapshot is not an independent backup either; it sits on the same datastore and is lost with it.
The key
The archive is encrypted with secrets/backup.key and signed with the same key. The key is created during the first installation, is not inside the backup, and cannot be regenerated from anywhere else.
There are two things you need to do:
- Copy
backup.keyto a safe place off the server, a password vault for example. If you put it where the backups are, the encryption protects nothing. - Set up an off-server copy of the backups. In a default installation the backups are only on the panel server's disk.
BACKUP_COPY_TOin.envtakes an rsync target; the signature file is copied along.
The signature has a limit: it catches a backup that was altered or put back under another name, but it does not prevent an unaltered old backup from being put back under its own name, nor backups from being deleted. Keep backups in a store you trust, and do not rename them.
Restore
Run the command without the confirmation first; it only checks and says what would change:
sudo sh scripts/install.sh restore backups/FILE.tar.encA confirmed run (--yes-replace-data) first checks that the database answers and that the dump can be read, then takes a safety backup of the current data, stops the services and replaces the database entirely with the one in the backup. Before it is checked, the archive is copied to a folder only root can open; the temporary folder needs free space of the archive's size.
What the hold is for
A restored database is older than the hypervisors, the payment providers and the people it describes. An operation left open in the backup, an unpaid order or a scheduled deletion can damage a server that has since been paid for and built. If the panel acted on its own on old records, it could delete a customer's running server.
So a restore does two things. It signs everyone out, because someone who signed out after the backup must not be signed in again. And it puts the panel on hold. While the hold lasts:
- the worker starts no operation;
- power and snapshot schedules and scheduled deletions do not run;
- collection does not run;
- the consistency check only writes findings and does not touch server records;
- API routes that start an operation answer
409 RESTORE_HOLD.
Signing in, viewing pages and the settings work. Staff with the system.update permission are notified, and staff see a banner at the top of every page.
Before you release the hold, look at what has changed since the backup: servers created or deleted, payments made, API tokens deleted, bans and role changes. Then release it from the banner under Administration → System or with this command:
docker compose exec api node dist/cli.js release-restore-holdIf you are restoring a backup taken a moment ago, right after a failed update for example, --no-hold skips the hold.
What has not been tried
The logic of the hold notification is covered by an integration test, but nobody has checked that the e-mail arrives after a real restore. In the end-to-end test the hold state is fed to the page from outside. The installer's backup and update paths were not re-read line by line in the security review. Try a restore once on an empty server before you really need one.
Sources: docs/operations.md, docs/updates.md (Turkish).
