A lifeboat for every hull

The git remote was a comfort, not a backup

A machine does not warn you before it dies. What survives is whatever you had already put somewhere else.

Every project in the fleet has a git remote. So if Caspar lost its disk tonight, I could clone all the code back onto a fresh box by morning and feel like I had lost nothing. That is the comfort. It is also not true.

The code was never the part that hurts. The part that hurts is the state that only exists on the running machine. A database with real customer records in it. A folder of files people uploaded. A directory of plain text that is the only map I have of how the whole fleet fits together. None of that is in git. It sits on one disk, and if the disk goes, so does it.

What did I actually have in place? A scatter. One project ran a nightly job that dumped its database to a folder on the same machine as the database, which is tidy and saves you from a bad migration but not from a dead drive. Another had a script meant to mirror code to private repositories twice a day. When I finally read its log it said Successfully backed up 0 out of 5, and had been saying that for weeks, because a token had expired and nobody was watching. On the NAS there was a folder with two tarballs in it from a migration back in March. Nothing anywhere could answer the one question that matters: is there a recent, working copy of everything, somewhere that is not the machine it came from?

That was the point to stop. Backups are not a feature you bolt onto a project. They are a discipline you hold the whole fleet to, the same as holding it to having a spec or a git remote. And I already run the thing that holds the fleet to standards and keeps an eye on every machine. It is called Control Room. This belonged there.

So I built it into Control Room

One day of work, three parts. Control Room owns a small config file per project that says what needs saving and where it goes. A separate program, the agent, reads that config, holds the actual credentials, and does the dumping. A screen in Control Room reads back what the agent did, so at a glance I can see whether every hull has a lifeboat and whether it still floats.

The rest of this is those parts, and the two machines that nearly broke the whole thing.

Git already carries the code and the config. Everything else lives on one disk until something copies it off.

What actually needs saving

Why it is here. “Back up the project” is too vague to act on. You have to name every piece of state, one at a time, and decide what it is: already covered, needs a database dump, needs a file archive, or genuinely does not matter.

How it works. Each project gets a backups.yml next to its record in Control Room. It lists stores. A store is one thing to save: a Postgres database, a directory, a Docker volume. For each one: how often, how many copies to keep, whether to encrypt.

Going through the six that matter:

cgburchell keeps its blog posts and contact-form submissions as flat JSON in one folder and its uploaded images in another. No database. Two file archives.
playtopia has two Postgres databases and two data volumes the game services treat as their own store. Four.
gp-forms has one Postgres database with real customer records in it. One store, and the only one that gets encrypted.
Control Room’s own data is a directory of Markdown and YAML. That one I handled differently: I made it a git repository with a private remote, so it versions itself and pushes on every change. Its backups.yml just says method: git and the agent leaves it alone.

The output of this part is a manifest. Nothing runs yet. But every project can now answer “what would I need to stand this back up”, which it could not before.

Every stateful thing lands in one of four buckets. The method follows from the bucket.

Where the copies go

Why it is here. A backup on the same machine as the thing it backs up is not a backup, it is a convenience. The copy has to survive the machine.

How it works. There is a Synology NAS on the same network, already mounted on Caspar because Jellyfin streams media off it. It has 97 terabytes free. The backups go into a Backups folder on it.

The model is push. The agent runs on Caspar and writes each archive straight to the mounted folder. Nothing runs on the NAS. The one trap with a network mount is that when it drops, the mount point quietly becomes an empty directory on the local disk, and a job that does not check will write gigabytes into it while reporting success, filling the wrong disk and saving nothing. So before every run the agent checks that the mount is live and that a sentinel file it expects is sitting there. If either is missing it stops and says so. It is the same instinct as checking the lifeboat is tied to the davit before you lower anyone into it.

Caspar's agent writes to the mounted NAS folder. The two remote projects are pulled the same way. The guard runs first, every time.

How each thing gets dumped

Why it is here. A database and a folder of files are not saved the same way. A live database copied file by file is a corrupt database. It has to come out through the engine.

How it works.

Postgres: pg_dump --format=custom. Custom format because it compresses, and because pg_restore --list can read the table of contents out of it without restoring anything, which is how I check a dump is real.
Files and volumes: tar piped through zstd. One stream, so it is internally consistent, and zstd because it is fast with good ratios.
Then, for gp-forms only, the archive goes through age before it lands.

The encryption is worth a paragraph. gp-forms holds the only customer data in the fleet, and “private network” is not a security model. So its dump is encrypted with an age keypair, and the point is the split. The agent only ever holds the public key, which can encrypt and cannot decrypt. The private key, the one that can actually read a gp-forms backup, is the restore key. It lives in Control Room’s data, kept out of the git repository, and I have a copy in a password manager. If the agent’s machine is compromised, the attacker can write new backups. They cannot read the old ones. It works like a postbox: the slot takes letters from anyone, only the key opens it.

One pipeline. The age step is there for gp-forms only, and the agent never holds the key that reverses it.

The two machines I could not reach

Why it is here. Four of the six projects run on Caspar, where the agent runs, so it dumps them directly. Two run elsewhere: gp-forms on one production box, sportsball-coach on another. Those had to be dumped across the network.

How it works. The plan was a locked-down SSH key on each remote box that could run one command and nothing else: a script that dumps its database to standard output. I have done this before for the read-only monitoring key, so the shape was familiar.

Then it did not work. The dump script uses docker exec to reach the database inside its container, and docker exec failed on both machines:

Could not check if docker-default AppArmor profile was loaded:
open /sys/kernel/security/apparmor/profiles: permission denied

Both boxes are themselves unprivileged containers. Inside one, docker exec cannot set up the security profile it wants for the process it is about to start, so it refuses. Even as root.

The fix was nsenter. docker exec is a convenience: find the target container’s process, step into its namespaces, run your command there. nsenter does the stepping-in directly, without the wrapper and its AppArmor check. So the dump script became “find the Postgres container’s process id, nsenter into its mount and process namespaces, run pg_dump from inside”. It reaches the database through the container’s own local socket, so it needs no password at all.

The key that runs this script is restricted to it and nothing else. Presenting the key runs the dump. It cannot open a shell, forward a port, or read a file. I checked by asking it for /etc/shadow. It returned the bytes of a database dump.

docker exec could not load an AppArmor profile it did not have the rights to read. nsenter steps into the same namespaces without that check.

Config here, credentials there

Why it is here. Control Room is a web application behind a login. If I put database passwords and SSH keys into it so it could run backups itself, then a bug in the dashboard is a bug that reads every database in the fleet. That is a bad trade for a convenience.

How it works. The work is split. Control Room owns the config, the schedule, and the log, and renders the screens. It has no credentials. When I click “run backup now” on a project, all it does is write a small file into a requests folder.

The agent is a separate program on Caspar, on a timer. It reads the same config, holds the credentials, does the dumps. Once a minute it also checks the requests folder, and if it finds one it runs that project’s backup out of schedule and deletes the file.

So the dashboard can trigger a backup, schedule one, and show me everything about it, while the process that answers web requests never holds a secret. An attacker who gets into it can change when a backup runs. They cannot run one, and they cannot read one. It is the bridge and the engine room: the order comes from one place, the throttle is held in another.

The dashboard rings for a backup. The agent holds the throttle. They meet at a request file and a set of receipts.

Seeing it

Why it is here. A backup whose state you cannot see is one you find out about at the worst possible time.

How it works. After every store it saves, the agent writes a receipt: a small file with the timestamp, the size, a checksum, and whether it worked. Control Room reads those receipts the same way it reads everything else, live, every time the page loads.

Two views. Each project’s page has a Backups panel: every store, when it last ran, how big it was, whether that run is fresh or has gone stale, a lock next to the encrypted one, and a button to run it now. And there is a fleet screen that puts every project in one table, green where the last run is current, amber where it has gone quiet past its schedule, red where it failed.

It is also a standard now, the same as having a git remote. A project that has opted into backups and whose last run failed shows a failing check on the standards board, next to the ones that are missing a spec.

The agent leaves a receipt per store. Control Room turns a folder of receipts into a per-project panel, a fleet table, and one more standards check.

What is verified vs assumed

Verified. All six projects back up. The two remote databases dump over the restricted key, and I have restored both into a throwaway container to prove the dumps are real: gp-forms decrypts and lists a hundred tables, sportsball-coach lists forty-six. A full run of the fleet saves eight stores with no failures in about twenty seconds. The “run backup now” button writes a request and the agent picks it up inside two minutes. Unmounting the NAS makes the next run stop instead of writing to the local disk.

Assumed, and the next job. That an untested restore counts. I have listed the dumps. I have not stood a whole project back up from one on a fresh machine, and until I have, “restorable” is a hope with good evidence under it. The next piece is a written recovery procedure per project and a monthly job that restores each database into a scratch container and checks a row count. There is also the matter of the secrets themselves, the .env files and keys that are correctly in no repository and so are currently backed up nowhere. Same shape as everything here: name it, dump it, encrypt it, check it.