Backup

a copy of data against loss, and the making of it

What it is

A backup is what the data gets restored from when the original is deleted, corrupted or lost along with the machine that held it, so it has to live somewhere those events do not reach. The NIST glossary defines it as a copy of files and programs whose purpose is to make recovery easier. The noun, plural backups, names the copies, and the verb, written as two words, names the act: a nightly job backs up the database and leaves a backup behind.

Neighbouring things get mistaken for it. A replica receives every change within seconds, which protects against a dead disk and copies a mistaken DELETE just as faithfully. A snapshot freezes a volume at one moment, and becomes a backup only once it is stored where the original’s failures cannot follow. A backup is defined by history and separation: it returns the data as it stood at an earlier point, from a place that survived whatever took the original.

Databases make this concrete. The PostgreSQL manual describes three fundamentally different approaches. An SQL dump, made with pg_dump, writes out the statements that rebuild the database, as a consistent snapshot of the moment the dump began. A file system level backup copies the data directory itself. Continuous archiving combines one such base backup with the write-ahead log (WAL) the server produces afterwards, and replaying that log can restore the database to any moment since the base backup. That ability is called point-in-time recovery.

Why it matters

Two numbers turn a general wish to be safe into a design. NIST defines the recovery point objective, RPO, as the point in time to which data must be recovered after an outage: in practice, how much recent work the business can afford to lose. The recovery time objective, RTO, is how long a system can stay in recovery before the outage harms what the organization is there to do.

The two pull on different parts of the design. RPO is set by how often a copy is taken: a nightly dump can lose a day of writes, while WAL archiving shrinks the loss to the last few moments of log still on their way to the archive. RTO is set by how long a restore takes. The PostgreSQL manual notes that the server replays every WAL segment since the last base backup, which can take a while when that backup is old, so how often to take base backups is an RTO decision as much as a storage one.

Both numbers are claims about a restore, which is why a backup is judged by restoring it. A job that exits with status zero shows that bytes were written somewhere. Whether they rebuild a working system inside the RTO is answered by restoring on a schedule, somewhere other than production, and comparing what comes back.

Separation is the other half. A copy in the same account, under the same credentials, shares the failures it is meant to survive: a closed account, a stolen key, a script run against the wrong target. The useful question for any copy is which single event would take it out with the original.

What it looks like in practice

At the smallest scale, everything one company holds sat on a single cloud instance, and the provider’s snapshot of it was the entire recovery position: in the same account as the machine, carrying no encryption anyone here controlled, and never restored from. Its replacement is a systemd timer that takes a database dump, sends an encrypted, deduplicated restic snapshot to storage at a different provider over SFTP, prunes old snapshots on a retention policy and runs an integrity check. A dead-man’s switch is pinged only on success, so a silent run is what raises the alarm. A separate monthly restore drill pulls a known file back out of the repository and compares it, so the claim that the company can lose the host and get its data back rests on a restore that ran last month rather than a backup that ran last night.

At a larger scale, a hundred critical data backups were automated with Barman and Google Cloud for a product that lives on its GIS data. Barman handled PostgreSQL, Google Cloud held the offsite copies, and Bash and Python orchestrated and validated the runs. Integrity checks confirmed that each backup was good rather than merely present.

On a third product the work was to define the recoverable position instead of assuming it. Backup retention lives in the Bicep template beside the database, so point-in-time restore is a property of the template rather than a setting someone once clicked in a portal, and the environment around it is code too. The audit wrote the gaps down: geo-redundancy is switched off, and the RTO is proposed rather than measured, with the drill that would measure it described and waiting to run.

Where it goes wrong

Backup problems tend to stay hidden until a restore is needed, so each is worth looking for on a quiet day.

A file copy of a running database restores as a broken cluster. Copying PostgreSQL’s data directory with tar while the server runs collects files from different moments, and the manual states that the server must be shut down to get a usable backup this way. A filesystem snapshot taken at one atomic instant does work if it includes the WAL, which the server replays on start as it would after a crash; with data and WAL on different filesystems, a simultaneous snapshot may be impossible. pg_dump, or a base backup for continuous archiving, gives a consistent copy while the server keeps running.

A gap in the WAL archive ends point-in-time recovery at the gap. Recovery needs an unbroken sequence of archived WAL files reaching back to the start of the base backup. When the archive command keeps failing, PostgreSQL keeps retrying and WAL piles up in pg_wal/, so the first symptom is often a filling disk. The manual’s advice is to set up and test archiving before the first base backup, and to monitor it for delays.

The job stops, and the silence looks like success. A timer job that breaks writes its error to a log nobody reads, and each quiet day passes for a good one. Monitoring has to treat silence as the alarm: a check-in sent only after a successful run, to a service that pages when it is late, catches a job that has stopped as well as one that fails loudly.

The key goes missing, or arrives changed. Encryption moves the whole risk onto the key, and the restic documentation is blunt: losing the password means the data is irrecoverably lost. The key has to live somewhere that survives the machine. In the restic setup above, the change could have come from the supervisor itself: it interprets escape sequences in environment values, so a passphrase with a backslash, handed over that way, would have reached restic as different bytes from the ones the repository was created with, and without a word of warning. Reading it from a file closes that route.

Retention removes the copy that was needed. Corruption is often noticed weeks after it happened, and a policy that keeps seven daily copies has by then replaced every clean copy with a damaged one. A tiered policy keeps a long tail at low cost, and restic’s --dry-run shows what a policy would remove before anything goes:

restic forget --keep-daily 7 --keep-weekly 5 --keep-monthly 12 \
  --prune --dry-run

forget removes snapshots and prune removes the data only they referenced, so after a prune the old copies are gone for good.

The integrity check passes and the restore still breaks. By default, restic check verifies the repository’s structure without reading every pack file, since that means downloading all of them; --read-data reads everything, and --read-data-subset=1/5 reads one fifth, so five runs from 1/5 to 5/5 cover the lot. A full read proves only that the stored bytes are intact. Whether they rebuild a working database, with the right roles and settings, is something a restore shows.

Further reading

Glossary