{"canonical_url":"https://engineer.company/glossary/downtime/","content_markdown":"# Downtime\n\n## What it is\n\nDowntime is time during which a service is out of action for the people who\ndepend on it. Its other names are outage and unavailability; its opposite is\nuptime, and the share of time a service is up is its availability, usually\nquoted as a percentage over a month or a year.\n\nThe clean case is total: the site does not load, the API refuses connections,\nthe database is down. The common case is messier. A checkout that times out\nwhile the catalogue works, a login page that loads but rejects every password,\nan API answering one request in three: each is downtime for whoever needs that\npath, even though a health check pointed at the home page would report green.\nA useful definition counts what users can do, not whether a process is running.\n\nDowntime comes in two kinds. Unplanned downtime is the outage nobody chose: a\ndisk, a bad deploy, an expired certificate, a dependency that went down first.\nPlanned downtime is chosen in advance, usually inside a maintenance window, to\nupgrade, migrate or move something that cannot change while it serves traffic.\nPlanned downtime is easier to live with, since users can be warned, but it still\ncounts against availability unless the agreement explicitly excludes it.\n\nHTTP has a vocabulary for the moment itself. RFC 9110 defines the 5xx class of\nstatus codes for a server that knows it has erred or cannot perform what was\nasked. Three of them mean different things during an outage. 502 Bad\nGateway says a proxy got an invalid answer from the server behind it. 504\nGateway Timeout says the proxy waited too long for one. And 503 Service\nUnavailable is the one to send on purpose: the specification ties it to a\ntemporary overload or scheduled maintenance, and lets the server say when to\ncome back:\n\n```http\nHTTP/1.1 503 Service Unavailable\nRetry-After: 120\n```\n\nThat tells browsers, API clients and search engine crawlers the condition is\ntemporary and suggests waiting two minutes, which is a very different message\nfrom a broken page served with a success code.\n\n## Why it matters\n\nAvailability targets are written in nines, and the nines translate into time.\nOver a 365-day year, which is 8,760 hours, each target allows this much\ndowntime:\n\n- 99%: 87.6 hours a year, about three and a half days;\n- 99.9%: 8.76 hours a year, or 43.2 minutes in a 30-day month;\n- 99.95%: 4.38 hours a year, or 21.6 minutes in a 30-day month;\n- 99.99%: 52.56 minutes a year;\n- 99.999%: about 5.26 minutes a year.\n\nEach extra nine cuts the allowance by ten, and the cost of earning it rises\nsteeply. At 99.9% a person can be woken, log in, find the problem and fix it\ninside the monthly budget. At 99.99% the whole year allows less than an hour,\nwhich leaves room for perhaps one incident, so recovery has to be automatic.\nAt 99.999% even a routine restart has to happen without users noticing.\n\nDependencies multiply. A request that needs a load balancer, an application and\na database, each available 99.9% of the time and failing independently, is\nserved only when all three are up: 0.999³, or about 99.7%, roughly 26 hours of\ndowntime a year instead of 8.76. A service cannot be more available than the\nchain of things it needs, unless it removes links from the chain or duplicates\nthem.\n\nPlanning starts from the business side. NIST's contingency planning guide for\nfederal systems, SP 800-34, names three numbers worth settling before anything\nfails. The maximum tolerable downtime (MTD) is the total outage the owner of\nthe system will accept. The recovery time objective (RTO) is how long a system\ncan stay down before the impact becomes unacceptable, and it has to be shorter\nthan the MTD. The recovery point objective (RPO) is how far back in time the\ndata may be restored from the last backup, so it measures tolerable data loss\nrather than time offline. With those three written down, a backup schedule and\na failover design are answers to a question rather than guesses.\n\n## What it looks like in practice\n\nOn an\n[IPTV and OTT streaming platform with customers in China, the US and Germany](/portfolio/ran-24-7-support-for-1000-iptv-ott-servers/),\nevery hour was someone's prime time, so a quiet hour for maintenance did not\nexist. Support ran around the clock for roughly a thousand servers plus a good\nnumber of client-owned systems. Much of the work was noticing something\ndrifting before it became an outage, because on a live stream an outage is\nsomeone's television stopping. The platform stayed continuously available for\nits global audience.\n\nAt a mid-sized company,\n[ten PostgreSQL and SQL Server database servers ran at 99.9% uptime](/portfolio/ran-10-postgresql-and-ms-sql-servers-on-ubuntu/),\nwhich by the arithmetic above is an allowance of 8.76 hours a year each. Behind\nthe figure were automated backups with `pg_dump` and SQL Server Agent jobs,\nwith retention policies and offsite storage, monitoring in Prometheus and\nGrafana, and disaster recovery procedures documented for point-in-time restores\nand failover.\n\nAnd on a fleet of\n[thirty Ubuntu servers whose setup had grown by accident](/portfolio/managed-30-ubuntu-linux-vps-instances/),\nthe disaster recovery side was rebuilt with backups and restore procedures that\nwere actually tested. Network configuration became one hardened baseline across\nevery instance. The downtime risk dropped, with recovery paths that had been\ntested before anyone needed them.\n\n## Where it goes wrong\n\n**The monitor sits inside the outage.** A health check that runs on the same\nserver, or in the same data centre, keeps reporting green while users meet an\nexpired TLS certificate, a broken DNS record or a load balancer pointing at\nnothing. Probe from outside, from more than one place, along the path a real\nuser takes, and alert on what that probe sees.\n\n**The error page says 200.** A framework that renders a friendly error page with\na success status code hides the outage from monitoring, from clients that would\nretry, and from crawlers, which may index the error page in place of the real\none. Errors should carry an error status, and planned maintenance should\nanswer 503 with a Retry-After header.\n\n**The backup was never restored.** A backup proves only that bytes were copied.\nUntil a restore has been run, timed and checked against the data, the recovery\ntime is a guess, and it is often far longer than the RTO on paper. Restore on a\nschedule, into a separate environment, and write down how long it took. The\nsame applies to failover: a standby that has never taken over is a hope, and rehearsing failover under calm conditions helps expose problems before an\nincident.\n\n**Releases cause the outages they were meant to prevent.** A deploy that stops\nthe old version before the new one is ready leaves a gap on every release. A\nschema change that rewrites a large table under an exclusive lock blocks every\nquery against it until it finishes. Roll out new instances behind a readiness\ncheck before the old ones stop, split schema changes into an expand step and a\nlater contract step so old and new code both work in between, and on Kubernetes\nset a PodDisruptionBudget so that a node drain cannot take every replica down\nat once.\n\n## Further reading\n\n- [RFC 9110: HTTP Semantics, section 15.6](https://www.rfc-editor.org/rfc/rfc9110#section-15.6)\n  — the 5xx status codes, including 503 and its Retry-After hint.\n- [NIST SP 800-34 Rev. 1: Contingency Planning Guide for Federal Information Systems](https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final)\n  — where maximum tolerable downtime, RTO and RPO are defined.\n- [PostgreSQL: High Availability, Load Balancing, and Replication](https://www.postgresql.org/docs/current/high-availability.html)\n  — standby servers and failover, as the database's own manual describes them.\n- [PostgreSQL: Continuous Archiving and Point-in-Time Recovery](https://www.postgresql.org/docs/current/continuous-archiving.html)\n  — the base backup and WAL archive that a point-in-time restore depends on.\n- [Kubernetes: Disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)\n  — voluntary disruptions and the PodDisruptionBudgets that limit them.\n\n\u003chttps://engineer.company/glossary/downtime/\u003e\n","date_modified":"2026-11-12T05:34:23Z","description":"What counts as downtime, what 99.9% uptime allows in hours per year, how planned and unplanned outages differ, and how restore tests help assess recovery time.","formats":{"gemtext":"https://engineer.company/glossary/downtime/index.gmi","gopher":"https://engineer.company/glossary/downtime/gophermap","html":"https://engineer.company/glossary/downtime/","json":"https://engineer.company/glossary/downtime/index.json","markdown":"https://engineer.company/glossary/downtime/index.md","pdf":"https://engineer.company/glossary/downtime/index.pdf"},"kind":"page","language":"en","title":"What is downtime?","url":"https://engineer.company/glossary/downtime/","version":1}