{"canonical_url":"https://engineer.company/glossary/disaster-recovery/","content_markdown":"# Disaster recovery\n\n## What it is\n\nDisaster recovery is the part of contingency planning that covers the worst\ncase: the place a system runs in has been lost or is out of reach for a long\ntime, and the system has to be rebuilt somewhere else from copies. NIST's\nContingency Planning Guide for Federal Information Systems, SP 800-34 Revision 1\nfrom 2010, defines the disaster recovery plan as a written plan for recovering\none or more information systems at an alternate facility after a major hardware\nor software failure or the destruction of facilities. Its scope is narrow. The\nguide reserves it for major disruptions, usually physical ones, that deny access\nto the primary facility for an extended period, and it covers only the\ndisruptions that require relocation.\n\nThat narrowness sets it apart from its neighbours. An information system\ncontingency plan, in the guide's vocabulary, restores a single system wherever\nit happens to stand, whether after a failed disk or a corrupted database. The\ndisaster recovery plan is tied to a site: it moves operations to the alternate\nlocation, and from there each system is brought up by its own contingency plan.\nA plan for one database and a plan for losing a whole data centre, cloud region\nor provider account are different documents with different owners, and confusing\nthe two is how a company ends up with neither.\n\nThree measures turn the plan from an intention into something that can be\nchecked. Maximum tolerable downtime (MTD) is the total length of outage the\nsystem's owner is willing to accept. The recovery time objective (RTO) is the\nlongest a system resource can stay unavailable before the damage to other\nsystems and to the business becomes unacceptable, and because interrupted work\nalso has to be reprocessed afterwards, the guide says the RTO must normally be\nshorter than the MTD. The recovery point objective (RPO) looks backwards: it is\nthe point in time before the outage to which data can be recovered from the most\nrecent backup copy, and it expresses how much lost data the business can live\nwith. Put simply, the RTO is how long you are down and the RPO is how much work\nyou lose.\n\nThe alternate site comes in grades. A cold site offers space and infrastructure\nonly, a warm site is partly equipped, and a hot site is fully equipped and\nstaffed. The guide adds mobile sites, which in many cases can be delivered\nwithin 24 hours before setup begins, and mirrored sites, the most expensive\noption and the one it credits with \"virtually 100 percent availability\". Cold\nsites are the cheapest to maintain and may take substantial time to equip. A\nsecond cloud region or a second provider plays the same role for most software\ntoday, and the trade-off between what stands ready and what it costs to keep it\nready is the same.\n\n## Why it matters\n\nA backup is a copy, and a copy on its own runs nothing. Getting from the copy to\na working system also takes the software that reads it, the keys that decrypt\nit, servers and a network to run it on, the configuration that ties them\ntogether, and someone who knows the order to do it in. Any of those that existed\nonly in the lost site was lost with it. Disaster recovery is the discipline of\nmaking sure that all of them survive the same event, and that the path from copy\nto service has been walked before anyone needs it.\n\nRecoverable data is only as recent as the last copy that restores, whatever RPO the backup schedule is meant to meet. A nightly job that has failed silently for a week leaves a week’s data at risk, whatever the policy document says. NIST's guide makes the same point about tape,\nrecommending that backups be tested regularly to confirm that the data is stored\ncorrectly and that the files can be retrieved without errors or loss. A green\nstatus light on a backup job proves that something was written. Only a restore\nproves it can be read back.\n\nThe plan also decays. People leave, servers are replaced, providers redesign\ntheir consoles, and the passwords in the runbook expire. Of the seven steps NIST\nlays out for contingency planning, the last two are testing, training and\nexercises, and plan maintenance; the guide wants the plan to be a living\ndocument, reviewed at least once a year and whenever the system or the resources\nused for recovery change significantly. A plan that has never been rehearsed\ntends to fail on details nobody thought to write down: the order the services\nmust start in, a DNS record with a day-long time to live, the one credential\nthat was stored on the machine that is gone.\n\nFor software teams, the disaster is often an account rather than a building. A\ndeleted cloud project, a compromised administrator login, a suspended provider\naccount or a destructive command run against production removes access to the\nsystem as thoroughly as a flood does. Standby servers are no protection against\nthe last of these, since a standby applies a mistaken `DROP TABLE` within\nmoments, exactly as it applies every other change.\n\n## What it looks like in practice\n\nFor a PostgreSQL database, a recovery point far shorter than the gap between two\nbackups comes from continuous archiving of the write-ahead log (WAL) on top of a\nbase backup. The server copies each finished WAL segment somewhere safe, and a\nrestore replays those segments onto the base backup up to any chosen moment\nsince the base backup was taken:\n\n```ini\n# postgresql.conf on the protected server\nwal_level = replica\narchive_mode = on\narchive_command = 'test ! -f /mnt/archive/%f \u0026\u0026 cp %p /mnt/archive/%f'\n```\n\nTo restore, a base backup is unpacked into a fresh data directory, an empty file\nnamed `recovery.signal` is placed in it, and the recovery settings say where the\narchive is and where to stop:\n\n```ini\nrestore_command = 'cp /mnt/archive/%f %p'\nrecovery_target_time = '2026-03-10 17:14:00+01'\nrecovery_target_action = 'pause'\n```\n\nThe PostgreSQL documentation tells the story these settings exist for: someone\ndrops a critical table at 5:15 on a Tuesday evening, notices on Wednesday at\nnoon, and restores to 5:14. Pausing is the default once the target is reached,\nso that queries can confirm it is the right moment before the server is promoted\nto accept connections; if it is not, the server is shut down, given a later\ntarget and started again. The chosen recovery time must be later than the\ncompleted base backup, and the documentation advises setting up and testing the\narchiving before the first base backup is taken. When recovery completes, the\nserver removes `recovery.signal` so that it does not enter recovery again by\naccident.\n\nEngineer ApS offers [backup and disaster recovery](/services/backup-recovery/)\nas a service, described as backups that are tested rather than assumed and a\nwritten plan for the morning a server does not come back. Two portfolio pages\nshow what that means. In\n[backups and disaster recovery as code](/portfolio/implemented-backups-and-disaster-recovery-as-code/),\nfrom 2025, backup retention was set in Bicep beside the database, so\npoint-in-time restore is part of the template instead of a setting someone once\nclicked in a portal, and the environment around the database was defined as code\ntoo, because a database restored into an environment rebuilt from memory is\nhardly recovered. The recovery position was then audited and written down with\nits gaps named: geo-redundancy was switched off, and the recovery time objective\nwas proposed rather than measured, since no drill had been run.\n\nThe company's\n[backup and restore-test automation](/portfolio/backups-that-have-actually-been-restored/)\nwas designed to extend recovery beyond a provider snapshot. An Ansible role installs restic\nscripts and [systemd](/glossary/#systemd) units for a database dump where configured, encrypted\ndeduplicated snapshots to storage at another provider over SFTP, retention pruning and\nintegrity checks. A separate monthly workflow restores a known file to a scratch location\nand compares its bytes. Backup and restore have separate success signals, the passphrase\ncomes from a protected file, and timers depend on a configured repository. Those controls\nare implemented; a live recovery is a separate deliverable. The implementation record dated\n2026-10-27 says the timers were disabled without a repository and records no completed live\nrestore, so it supports the automation work rather than a recovery-time commitment or a\nclaim that the company can already lose the host safely.\n\n## Where it goes wrong\n\n**The backup has never been restored.** A job that reports success every night\nsays nothing about whether its output can be read, decrypted and loaded, and the\nfirst real restore is a poor moment to find out. Schedule restores the way you\nschedule backups: an automated check that pulls known data out and compares it,\nand a full restore into a clean environment often enough that the time it takes\nis a measured number.\n\n**The copy shares the original's fate.** A snapshot in the same cloud account, a\nbackup disk in the same rack or an archive bucket behind the same administrator\nlogin goes down with whatever takes the primary. NIST's guide asks for offsite\nstorage and alternate sites to be chosen for how unlikely they are to be hit by\nthe same disaster. Keep at least one copy with another provider or in another\naccount with separate credentials, and keep the decryption keys somewhere that\ndoes not depend on the systems they protect.\n\n**Only the data was planned for.** Restoring a database is a small part of\nbringing a service back if the servers, firewall rules, DNS, certificates and\nsecrets around it have to be recreated from memory. Define the environment in\nversion-controlled infrastructure as code, and make rebuilding it part of the\ndrill, not only loading the data.\n\n**The objectives are guesses.** An RTO written into a document without a drill\nbehind it is a proposal, and an RPO that assumes every backup succeeded is\noptimistic. Measure both in an exercise. NIST separates tabletop exercises,\nwhich are discussion only and deploy no equipment, from functional exercises\nthat simulate operations, up to full-scale ones that cover every element of the\nplan; the numbers come from the second kind.\n\n**A standby is counted as a backup.** A standby server or a geo-replicated\ndatabase protects against losing hardware, but it copies deletions, corruption\nand files encrypted by ransomware as faithfully as good writes. Keep\npoint-in-time copies that a mistake on the primary cannot reach, and keep them\nfor longer than it usually takes to notice a mistake.\n\n**The plan describes last year's system.** A runbook that lists retired servers,\npeople who have left and passwords that no longer work slows a recovery down\ninstead of guiding it. Review the plan after every significant change and after\nevery drill, record what the drill got wrong, and review the parts that change\nmost often, such as contact lists, more often still, as NIST also recommends.\n\n## Further reading\n\n- [SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems](https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final)\n  — the full guide: how it defines and scopes the DRP, the seven planning steps,\n  MTD, RTO and RPO, alternate sites and exercise types, with plan templates.\n- [Continuous Archiving and Point-in-Time Recovery (PITR)](https://www.postgresql.org/docs/18/continuous-archiving.html)\n  — PostgreSQL 18's chapter on WAL archiving, base backups and restoring to a\n  chosen moment.\n- [Write Ahead Log](https://www.postgresql.org/docs/18/runtime-config-wal.html)\n  — the configuration reference for `archive_command`, `restore_command` and the\n  recovery target settings.\n\n\u003chttps://engineer.company/glossary/disaster-recovery/\u003e\n","date_modified":"2026-11-12T05:34:23Z","description":"How a service is brought back somewhere else after losing its site or account, the targets that measure it, and why an untested plan is only a hope.","formats":{"gemtext":"https://engineer.company/glossary/disaster-recovery/index.gmi","gopher":"https://engineer.company/glossary/disaster-recovery/gophermap","html":"https://engineer.company/glossary/disaster-recovery/","json":"https://engineer.company/glossary/disaster-recovery/index.json","markdown":"https://engineer.company/glossary/disaster-recovery/index.md","pdf":"https://engineer.company/glossary/disaster-recovery/index.pdf"},"kind":"page","language":"en","title":"What is disaster recovery?","url":"https://engineer.company/glossary/disaster-recovery/","version":1}