Disaster recovery

What it is

Disaster recovery is the part of contingency planning that covers the worst case: the place a system runs in has been lost or is out of reach for a long time, and the system has to be rebuilt somewhere else from copies. NIST’s Contingency Planning Guide for Federal Information Systems, SP 800-34 Revision 1 from 2010, defines the disaster recovery plan as a written plan for recovering one or more information systems at an alternate facility after a major hardware or software failure or the destruction of facilities. Its scope is narrow. The guide reserves it for major disruptions, usually physical ones, that deny access to the primary facility for an extended period, and it covers only the disruptions that require relocation.

That narrowness sets it apart from its neighbours. An information system contingency plan, in the guide’s vocabulary, restores a single system wherever it happens to stand, whether after a failed disk or a corrupted database. The disaster recovery plan is tied to a site: it moves operations to the alternate location, and from there each system is brought up by its own contingency plan. A plan for one database and a plan for losing a whole data centre, cloud region or provider account are different documents with different owners, and confusing the two is how a company ends up with neither.

Three measures turn the plan from an intention into something that can be checked. Maximum tolerable downtime (MTD) is the total length of outage the system’s owner is willing to accept. The recovery time objective (RTO) is the longest a system resource can stay unavailable before the damage to other systems and to the business becomes unacceptable, and because interrupted work also has to be reprocessed afterwards, the guide says the RTO must normally be shorter than the MTD. The recovery point objective (RPO) looks backwards: it is the point in time before the outage to which data can be recovered from the most recent backup copy, and it expresses how much lost data the business can live with. Put simply, the RTO is how long you are down and the RPO is how much work you lose.

The alternate site comes in grades. A cold site offers space and infrastructure only, a warm site is partly equipped, and a hot site is fully equipped and staffed. The guide adds mobile sites, which in many cases can be delivered within 24 hours before setup begins, and mirrored sites, the most expensive option and the one it credits with “virtually 100 percent availability”. Cold sites are the cheapest to maintain and may take substantial time to equip. A second cloud region or a second provider plays the same role for most software today, and the trade-off between what stands ready and what it costs to keep it ready is the same.

Why it matters

A backup is a copy, and a copy on its own runs nothing. Getting from the copy to a working system also takes the software that reads it, the keys that decrypt it, servers and a network to run it on, the configuration that ties them together, and someone who knows the order to do it in. Any of those that existed only in the lost site was lost with it. Disaster recovery is the discipline of making sure that all of them survive the same event, and that the path from copy to service has been walked before anyone needs it.

The RPO is set by the last copy that restores, not by the backup schedule. A nightly job that has failed silently for a week leaves an RPO of a week, whatever the policy document says. NIST’s guide makes the same point about tape, recommending that backups be tested regularly to confirm that the data is stored correctly and that the files can be retrieved without errors or loss. A green status light on a backup job proves that something was written. Only a restore proves it can be read back.

The plan also decays. People leave, servers are replaced, providers redesign their consoles, and the passwords in the runbook expire. Of the seven steps NIST lays out for contingency planning, the last two are testing, training and exercises, and plan maintenance; the guide wants the plan to be a living document, reviewed at least once a year and whenever the system or the resources used for recovery change significantly. A plan that has never been rehearsed tends to fail on details nobody thought to write down: the order the services must start in, a DNS record with a day-long time to live, the one credential that was stored on the machine that is gone.

For software teams, the disaster is often an account rather than a building. A deleted cloud project, a compromised administrator login, a suspended provider account or a destructive command run against production removes access to the system as thoroughly as a flood does. Standby servers are no protection against the last of these, since a standby applies a mistaken DROP TABLE within moments, exactly as it applies every other change.

What it looks like in practice

For a PostgreSQL database, a recovery point far shorter than the gap between two backups comes from continuous archiving of the write-ahead log (WAL) on top of a base backup. The server copies each finished WAL segment somewhere safe, and a restore replays those segments onto the base backup up to any chosen moment since the base backup was taken:

# postgresql.conf on the protected server
wal_level = replica
archive_mode = on
archive_command = 'test ! -f /mnt/archive/%f && cp %p /mnt/archive/%f'

To restore, a base backup is unpacked into a fresh data directory, an empty file named recovery.signal is placed in it, and the recovery settings say where the archive is and where to stop:

restore_command = 'cp /mnt/archive/%f %p'
recovery_target_time = '2026-03-10 17:14:00+01'
recovery_target_action = 'pause'

The PostgreSQL documentation tells the story these settings exist for: someone drops a critical table at 5:15 on a Tuesday evening, notices on Wednesday at noon, and restores to 5:14. Pausing is the default once the target is reached, so that queries can confirm it is the right moment before the server is promoted to accept connections; if it is not, the server is shut down, given a later target and started again. The chosen recovery time must be later than the completed base backup, and the documentation advises setting up and testing the archiving before the first base backup is taken. When recovery completes, the server removes recovery.signal so that it does not enter recovery again by accident.

Engineer ApS offers backup and disaster recovery as a service, described as backups that are tested rather than assumed and a written plan for the morning a server does not come back. Two portfolio pages show what that means. In backups and disaster recovery as code, from 2025, backup retention was set in Bicep beside the database, so point-in-time restore is part of the template instead of a setting someone once clicked in a portal, and the environment around the database was defined as code too, because a database restored into an environment rebuilt from memory is hardly recovered. The recovery position was then audited and written down with its gaps named: geo-redundancy was switched off, and the recovery time objective was proposed rather than measured, since no drill had been run.

In backups that have actually been restored, from September 2026, the starting point was a company whose repositories, database and website sat on one cloud instance, with the provider’s snapshot as the entire recovery position: held in the same account as the machine, not encrypted by the company, and never once restored. The replacement runs on a systemd timer: a database dump where there is one, then an encrypted, deduplicated restic snapshot to storage at a different provider over SFTP, then a retention prune and an integrity check. A dead-man’s switch is pinged only when the run succeeds, so silence raises the alert. Every month a restore drill pulls a known file out of the repository and compares it. The page’s claim that the company can lose the host and get its data back rests on that monthly restore, not on last night’s backup.

Where it goes wrong

The backup has never been restored. A job that reports success every night says nothing about whether its output can be read, decrypted and loaded, and the first real restore is a poor moment to find out. Schedule restores the way you schedule backups: an automated check that pulls known data out and compares it, and a full restore into a clean environment often enough that the time it takes is a measured number.

The copy shares the original’s fate. A snapshot in the same cloud account, a backup disk in the same rack or an archive bucket behind the same administrator login goes down with whatever takes the primary. NIST’s guide asks for offsite storage and alternate sites to be chosen for how unlikely they are to be hit by the same disaster. Keep at least one copy with another provider or in another account with separate credentials, and keep the decryption keys somewhere that does not depend on the systems they protect.

Only the data was planned for. Restoring a database is a small part of bringing a service back if the servers, firewall rules, DNS, certificates and secrets around it have to be recreated from memory. Define the environment in version-controlled infrastructure as code, and make rebuilding it part of the drill, not only loading the data.

The objectives are guesses. An RTO written into a document without a drill behind it is a proposal, and an RPO that assumes every backup succeeded is optimistic. Measure both in an exercise. NIST separates tabletop exercises, which are discussion only and deploy no equipment, from functional exercises that simulate operations, up to full-scale ones that cover every element of the plan; the numbers come from the second kind.

A standby is counted as a backup. A standby server or a geo-replicated database protects against losing hardware, but it copies deletions, corruption and files encrypted by ransomware as faithfully as good writes. Keep point-in-time copies that a mistake on the primary cannot reach, and keep them for longer than it usually takes to notice a mistake.

The plan describes last year’s system. A runbook that lists retired servers, people who have left and passwords that no longer work slows a recovery down instead of guiding it. Review the plan after every significant change and after every drill, record what the drill got wrong, and review the parts that change most often, such as contact lists, more often still, as NIST also recommends.

Further reading

• SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems — the full guide: how it defines and scopes the DRP, the seven planning steps, MTD, RTO and RPO, alternate sites and exercise types, with plan templates.
• Continuous Archiving and Point-in-Time Recovery (PITR) — PostgreSQL 18’s chapter on WAL archiving, base backups and restoring to a chosen moment.
• Write Ahead Log — the configuration reference for archive_command, restore_command and the recovery target settings.