Measuring the hold before moving the cargo

Moving a project between anchorages used to mean a guess dressed up as a decision. SSH into the target host, run df -h and docker stats, eyeball what’s free, and hope the project’s actual appetite under load was smaller than what the snapshot showed. Half the time it was fine. The other half I found out after the move, when a host that looked roomy at 11pm was pegged at 90% RAM by the next afternoon, because the one number I’d checked was a single moment and the project’s real habit was spikier than that.

So the dashboard now keeps the numbers instead of a glance. A cron job samples RAM and CPU on every live-monitored host every five minutes and keeps fourteen days of it. A second job, once a day, measures disk per project rather than per host. And a move simulator sits under the capacity bars: pick a project and a target anchorage, and it tells you whether the hold has room, using those numbers instead of a glance at the quay.

Why a snapshot lies about a spiky hull

A single reading flatters or damns a project depending on when you happened to look. The capacity sampler fixes that by recording instead of glancing: every five minutes, on every host, RAM and CPU go into history, and fourteen days of it is kept. Once at least 24 hours exists for a project, the bars on the Servers page stop showing the live number and switch to its p95: the typical busy moment, not the best or worst one. Peak is still shown alongside it, and a project whose peak runs well clear of its p95 gets a mark in the legend, so a hull that’s usually calm but occasionally spikes hard doesn’t look identical to one that’s steady.

Containers get mapped to projects by name each time the page renders, the same discovery the rest of the dashboard already uses, so a container that joins a project later is counted under it for its whole recorded history, not just from the day it was tagged.

Weighing the hold, not just the deck

RAM and CPU are quick to sample because they’re a live number any Docker API call hands over. Disk isn’t: working out what belongs to which project means walking image layers, container writable layers, named volumes and bind-mount folders, and some of that is slow enough that it only runs once a day rather than every five minutes.

The split is specific rather than a rough percentage. Each project’s unique image layers count against it; layers two projects share get split evenly between them, since neither one alone caused the other’s base image to exist. Each container’s own writable layer counts fully. Named volumes split if more than one container uses them. Writable bind-mount folders on the root filesystem get measured with du, at low priority, since walking a folder tree is the slow part of the whole job; a folder the measuring user can’t read just counts low rather than failing the run, and a folder nested inside one already measured isn’t counted twice. Anything left over, images and volumes no running container uses, shows up separately as reclaimable, and everything else on the disk, shared layers, build cache, the OS itself, is honestly labelled other rather than folded into a project it has nothing to do with.

Both jobs are one script behind a lock file, so a slow disk walk on a busy day can’t start a second copy of itself on top of the first; they report themselves as scheduled jobs the same way backups do, so a sampler that’s stopped running shows up rather than just quietly going stale. On a remote host the disk measurement rides the same forced, read-only SSH command this fleet has used since a key with one job, just a new branch in the script that answers exactly one more request and still refuses everything it didn’t expect.

The move simulator

All of that exists to answer one question without a dry run: if this hull moved to that anchorage right now, would it fit? Pick a running project and a target server, and the simulator takes the project’s p95 share where there’s history (its live usage otherwise) and lays it onto the target without rescaling anything, the same size it is today, nothing trimmed to make the numbers work. RAM and disk, when disk has been measured, get checked against two lines: under 80% is a fit, over 80% is tight, over 100% doesn’t fit, with the shortfall spelled out rather than left for me to subtract. CPU rides along as advice only, since the host total behind it is a load-average estimate rather than a real reservation.

Fitting isn’t the whole question, so the simulator also flags the parts of a move that are more than docker compose up somewhere else: a database container that needs a real migration rather than a restart, named volumes and writable bind mounts that have to travel with the data, Traefik routing labels that assume the project lives at its current address, and a change in CPU architecture between the two hosts. None of those block the verdict. They’re printed alongside it, because a project can fit the hold perfectly and still need more than a copy-paste to actually move.

I still haven’t moved anything with it, verdict untested against a real migration. What’s verified is the numbers: a week of watching the Servers page against what docker stats and du report by hand, and they agree. Fair winds.