The key to the lifeboat was on the ship
A few weeks ago, in Launching the lifeboat, I wrote that every backup in the fleet now gets restored for real once a week, and that the keys and env files, saved nowhere before, now go off the box nightly, encrypted. Both of those are still true. This week the content planner joined the fleet, and writing up its backups is how I found the one key that sentence didn’t cover: the key that opens the backups themselves.
The planner lives out on a rented VPSVirtual Private Server — a slice of a machine you rent in a data centre, with its own public IP address. A few dollars a month gets you enough to run a small production app.: one PostgresPostgreSQL — a widely used database. If a project stores accounts, orders, posts, anything structured, it is probably in a Postgres container. database and one DockerA tool that packages an app together with everything it needs to run into a "container", so it runs the same on any machine and does not collide with anything else installed. volume holding every scheduled post and the tokens that let it post as me. It gets the same treatment as the rest of the fleet, so I won’t walk through it again. The main server pulls a dump and a media archive every night over two keys that can each do one job, seals them with ageA small, modern file-encryption tool. You encrypt to a public "recipient" string; only the matching secret identity can decrypt. Used here for the one backup with real customer data in it. and writes them to the NASNetwork-Attached Storage — a box of hard drives on your home network that other machines can write to. Or, honestly, any spare drive you copy backups onto., exactly as A lifeboat for every hull set up. The only new part was teaching the agent to pull a files store over an SSH alias. The first restore-testRestoring a backup somewhere harmless, like a throwaway database, to prove it really opens and holds what it should. came back with 87 tables and a media archive that unpacked cleanly.
Where the key lives
Decrypting needs the private half of the age key, and while noting the planner down in the recovery order I went to write where it was kept. It was kept on the main server. In a gitignored folder in the data repo, mode 600, exactly where I’d put it for the two projects that were already encrypted, and nowhere else. The nightly secrets bundle from the last note gathers up every .env file, the SSH keys and the mesh certificates into one encrypted archive on the NAS. I read its list of paths. That folder isn’t on it.
Why nothing went red
The backups page was green the whole time, and it was telling the truth. Every archive existed, every checksum matched, every weekly restore opened and counted its rows. The catch is where the test runs. It runs on the main server, and the main server is where the key is, so the one thing it can’t ever notice is that the key is only there.
So picture the bad night. the main server’s disk dies, or the house floods, or I fat-finger something as root. The NAS is fine and every lifeboat is sitting there, fourteen nights deep, sealed with a key that went down with the ship. For the planner that’s a weekend of reconnecting accounts. For the two older projects it’s real data I couldn’t get back.
Getting the key off the ship
Not all treasure is silver and gold, mate, and a private key is about the smallest treasure there is: 189 bytes. That’s why it’s easy to forget. The fix is just as small. Either the folder joins the secrets bundle, which is itself encrypted to a key I already keep somewhere off the ship, so one well-kept key opens everything; or each key gets a copy somewhere that isn’t the main server, like the password manager. I haven’t picked yet. At the time of writing it’s a warning in the dashboard’s Needs you list, which is at least more than it was a week ago. The test needs the same lesson: once a quarter it should run somewhere that isn’t the main server, with the key fetched from wherever the copy ends up.
Check where your keys sleep. Fair winds.
-x