Built encrypted off‑host backups on restic with retention pruning, an integrity check and a monthly automated restore drill, then audited the recovery position and wrote down the gaps rather than leaving them to be found during an incident.
Work carried out: 2025
Situation. Everything the company holds — the git repositories, the database, the site — sat on one cloud instance whose provider snapshot was the entire recovery position. A provider snapshot is a fine thing to have and a poor thing to rely on: it is in the same account as the machine it protects, it is not encrypted by anyone here, and nobody had ever restored from one.
Task. Backups had to be encrypted, off‑host, pruned on a retention policy, and — the part that is usually skipped — actually restored from, on a schedule, without a person remembering to do it.
Action. The backup runs on a systemd timer: a database dump where one exists, then an encrypted deduplicated snapshot to storage at a different provider over SFTP, then a retention prune, then an integrity check. A dead‑man’s switch is pinged on success only, which is the distinction that makes it an alarm rather than a log — a run that fails says nothing, and saying nothing is what raises the alert. Separately, a restore drill runs monthly: it pulls a known file out of the repository and compares it, so the thing being checked is a restore rather than a backup. The passphrase is written to a file the units read rather than passed through the environment, because the supervisor processes escapes in environment values and a passphrase containing a backslash would have been silently different from the one that created the repository. Afterwards the recovery position was audited and written up, and the remaining gaps were named in the document rather than left to be discovered during an incident.
Result. The company can lose the host and get its data back, and that sentence rests on a restore that ran last month rather than on a backup that ran last night. The audit’s most valuable output was the list of things still not covered, which is the part a green backup report is structurally unable to tell anyone.