Launching the lifeboat

What the backups note left undone

A backup you have never restored is a rumour.

An earlier note built the backup half of this: a job that dumps every database and every folder of files a dead machine would take with it, writes the copies to a box in another room, and encrypts the ones that hold anything sensitive. Since then I have also extended bosun-x so that each project’s backup and recovery state is visible on its project page, along with info like is there a current copy, when did it last verify, and the controls to run either on demand.

What it did not have was a single restore that had actually happened. I had asked each dump for its table of contents and it had answered. That is not the same as standing the database back up. Until you have done that, “restorable” is a hope with good evidence under it.

A restore that tests itself

So now it tests itself, once a week.

The job walks every current backup and, for each one:

checks the archive is still on the far box and its checksum still matches, because a copy can rot in place;
unpacks it: decrypt, then decompress with ;
for a database: starts a fresh, empty in a , runs a real restore into it, then counts the tables and the live rows;
for a folder of files: lists every entry and confirms the shape is intact;
writes the result where the dashboard reads it, so last verified restore sits next to last backup.

The container it restores into is built to be harmless: no name, no network, and it deletes itself when the check is done. It cannot be reached, cannot reach out, and cannot be mistaken for anything real. It lives about fifteen seconds.

Beware the Postgres image’s first boot. It starts, sets itself up, stops, and starts again for real. A restore fired into that first flicker lands in a database that is about to be wiped. The check waits until the container answers a trivial query three times running before it trusts it is the one that will stick.

Once a week, per backup: check the archive and its checksum, unpack it, restore it for real into a container that has no name and no network and deletes itself after, count what came back, show the result. Any failure shows red on the project page.

The last full run put nine stores back and lost none. The largest came back as a hundred and thirty-nine tables and ninety thousand rows; the smallest, eight tables and eighty. The encrypted one decrypts and restores like the rest. What a project’s page shows now is not backed up six hours ago. It is restored clean, and here is what came back.

That time I deleted what I was testing

The first version of this job did real damage.

It was meant to only ever touch its own throwaway containers. I edited it while its own was mid-run, and a stale copy executed a teardown that matched containers by name pattern rather than by label. It stopped and removed six of them, two of which were live production databases. The site they belonged to was half down for about eight hours while I restored them from the backups. That is the system working, but not like that.

The fleet scripts now run under rules that do not bend:

a container is started only through one helper that forces no name, no network, and a unique label;
a container is removed only through one helper that matches that label and nothing else, re-checked with an inspect before the delete. Never by name, never by image. No stop, kill, prune, or compose-down anywhere in these scripts, at all;
the dashboard’s own view of is read-only, with no method that can stop, remove, or start a thing;
after any change to one of these scripts, a check must pass that stands up decoy containers with tempting names and proves every one survives a full run.

A backup system you cannot trust around your running machines is not one. This is what it costs to be trusted.

The one button

The weekly test proves the archives are good. It does not help you at eleven at night when a bad change has mangled a real table and you want this database, from last night, now.

That is the most dangerous thing in the whole fleet, because it overwrites a database that is live. So it is behind four locks:

it will not run without a confirmation the interface supplies and a slipped command line does not;
its first act, always, is a fresh dump of the current database. If that dump fails, nothing else happens. That dump is the undo, and restoring from it puts things back;
the target has to be exactly the container named in that project’s config, not one that merely looks right;
restoring the latest backup re-checks its checksum first.

It works for a database running in a container on the main server. Two of the projects keep their database on a separate production machine of their own, and the button does not reach those yet. The dump already comes back over a locked-down key that can run one command and nothing else, and giving that key a matching restore command is the next job. Until then those two go back by hand, off the written steps.

A restore into the live database dumps the current contents first. If that dump fails the whole thing aborts. That pre-restore dump is an ordinary archive, and restoring from it is the undo.

The order, written down

Knowing each archive restores does not tell you how to bring a whole machine back. What has to come up before what. Which key opens which bundle. You work that out once, calmly, and write it down, because the day you need it you will be tired and it will be late.

The order is:

the private network first, because nothing else can talk until it is up;
then the secrets, because no service will start without them;
then the plain-text state, which is a clone, not a restore;
then the code;
then the databases and files;
then bring everything up;
then check one health endpoint, one login, one record you recognise, and write down that you did.

Each project also carries a short recovery file in its own repository, so the steps travel with the code.

There is one case the document is honest about not covering: the server and the box its backups sit on are in the same room, so a fire or a theft takes both. What would be left is the code in version control and whatever is in a password manager. Every database’s contents would be gone. That is an accepted risk, not a solved one.

The secrets, finally on the list

The env files, the keys, the certificates. Deliberately in no repository, and so the one kind of thing with no backup at all. Right not to commit them. Wrong to have exactly one copy.

They get the same treatment as everything else: name it, pack it, encrypt it, check it. A nightly job tars every env file, the SSH keys, the backup agent’s own keys, and the private-network certificate and its key, runs it through compression and then , and writes it beside the rest.

It opens with one identity, and that identity is the single thing that is in no backup anywhere. It would be sitting next to its own ciphertext. It is in a password manager and nowhere else. The weekly test unpacks that bundle too.

What holds and what doesn’t

Holds. Every backup restores, once a week, into a real database or a real listing, with the counts written down. The in-place restore has been run against a live database and the pre-restore dump brought it back. The secrets bundle has been unpacked start to finish. The test cannot harm a running container, and there is a decoy suite that proves it after every change.

Does not, yet. I have restored databases into machines that were still standing. I have not wiped a whole machine and rebuilt it from nothing with a clock running. Until I do, the recovery order is a plan I believe rather than one I have run. The two databases on their own machines still go back by hand. And a fire in the server room would still take every database with it.

It floats. I have not had to launch it in weather.