{"canonical_url":"https://engineer.company/glossary/high-availability/","content_markdown":"# High availability\n\n## What it is\n\nHigh availability is the property of a service that keeps answering when one of\nthe machines, disks, network links or processes behind it fails. It is usually\nbuilt as a cluster: several hosts that can each run the service, software that\nwatches them, and a procedure that moves the work from a failed host to a\nhealthy one. The ClusterLabs project, which maintains the open-source Pacemaker\nand Corosync cluster stack, describes such a cluster as managing services on\nseveral hosts, from starting them and checking their health to recovering them\nafter failure, \"to minimize downtime while preserving data integrity\".\n\nThe second half of that sentence matters as much as the first. A service that\ncomes back quickly with the wrong data has not been made highly available. It\nhas been made dangerous.\n\nPostgreSQL's documentation separates two reasons for running several database\nservers together. One is to let a second server take over quickly if the\nprimary fails, which it calls high availability. The other is to let several\ncomputers serve the same data, which it calls load balancing. The two often\nshare machinery, but they answer different questions: high availability is\nabout surviving a failure, while load balancing and scalability are about\ncarrying more work.\n\nAvailability itself is a number. The traditional measure is time-based: the\nshare of a period during which the service could be used. Google's SRE book\nexplains that for services spread around the world, where some part is nearly\nalways up somewhere, it measures the share of requests answered successfully\ninstead. Both are written as percentages and spoken of as a count of nines.\n\nEvery highly available design is assembled from the same few parts:\n\n- **Redundancy.** A spare for every part whose failure would stop the service:\n  a second host, a standby database, a second network path. Pacemaker supports\n  active/passive pairs, N+1 arrangements in which several active nodes share\n  one spare, and other layouts.\n- **Detection.** Something has to notice the failure. PostgreSQL's chapter on\n  failover describes a heartbeat between primary and standby that continually\n  checks the connection and whether the primary is still working.\n- **Failover.** Moving the work: promoting the standby, moving the service's IP\n  address, starting the process on another host.\n- **Fencing.** Making sure the failed node really has stopped before its work\n  starts somewhere else.\n- **Rebuilding the spare.** After a failover the system runs without redundancy\n  until a new spare is in place.\n\nHigh availability is neither a backup nor disaster recovery. It protects\nagainst the failure of a part while the site as a whole is still running. A\nbackup protects against losing or corrupting data, including corruption that a\ncluster faithfully copies to every node, and disaster recovery is the plan for\nlosing the whole site.\n\n## Why it matters\n\nEvery service has parts that will fail. Hardware breaks, kernels panic, a\nvirtual machine is deleted by mistake, a network partition cuts a healthy node\noff from the rest of the cluster. The Kubernetes documentation lists these as\ninvoluntary disruptions and sets them beside voluntary ones that somebody\nplans, such as draining a node for a repair or an upgrade. Without redundancy\neach of them is an outage, the planned ones included, because the only way to\npatch a single server is to take it down.\n\nThat second point is often the first benefit a team notices. PostgreSQL's\nfailover chapter calls regular switching between primary and standby useful:\nit lets each machine be taken down in turn for maintenance, and every switch\ntests that the failover mechanism will work when it is needed.\n\nThe nines translate into time. For a 365-day year (525,600 minutes) and a\n30-day month (43,200 minutes), the arithmetic is:\n\n```text\nAvailability   Allowed per year       Allowed per 30-day month\n99%            3.65 days              7.2 hours\n99.9%          8.76 hours             43.2 minutes\n99.95%         4.38 hours             21.6 minutes\n99.99%         52.56 minutes          4.32 minutes\n99.999%        about 5.26 minutes     about 26 seconds\n```\n\nEach extra nine cuts the allowance by a factor of ten, and the cost climbs\nfaster than that. The SRE book warns that one further step in reliability may\ncost a hundred times as much as the step before it, and that past a point users\ncannot tell the difference: someone on a phone that is itself only 99%\nreliable will not notice whether the service behind it runs at 99.99% or\n99.999%. The book therefore treats an availability target as both a floor and a\nceiling. A service far more available than it needs to be has spent money and\nengineering time that could have gone elsewhere.\n\nThat is why the target is a business decision, made service by service. A\nservice level objective writes the number down; high availability is the\nengineering that makes it reachable.\n\n## What it looks like in practice\n\nTake an invented booking service with three tiers: a web tier, an application\ntier and a PostgreSQL database, each on its own host. Every number below is\nmade up for the example. If each tier is up 99.9% of the time and the service\nneeds all three, their availabilities multiply. Doubling a tier works the other\nway round, because both copies have to be down at once:\n\n```text\nThree tiers in series, each 99.9% available:\n  0.999 × 0.999 × 0.999 ≈ 0.997      about 26 hours down a year\n\nOne tier doubled, if its two copies fail independently:\n  1 − 0.001 × 0.001 = 0.999999       about half a minute a year\n```\n\nThe second calculation rests on one word: independently. Two copies on the same\nhost, in the same rack, behind the same switch or in the same cloud zone share\na fate, and the arithmetic stops applying. The formula also assumes the switch\nfrom one copy to the other is instant and always works, which only testing can\nshow.\n\nOn Kubernetes, that is what topology spread constraints are for. The\ndocumentation's own motivating case is a workload that may shrink to two Pods:\nyou would not want both on the same node, where one failure takes the workload\noffline. A constraint in the Pod template spreads the replicas across zones:\n\n```yaml\ntopologySpreadConstraints:\n    - maxSkew: 1\n      topologyKey: topology.kubernetes.io/zone\n      whenUnsatisfiable: DoNotSchedule\n      labelSelector:\n          matchLabels:\n              app: booking\n```\n\nWith `DoNotSchedule`, the scheduler leaves a Pod unscheduled rather than place\nit where the zones would differ by more than one. Planned maintenance needs a\nsecond guard. A PodDisruptionBudget limits how many replicas of an application\nmay be down at once from voluntary disruptions. `kubectl drain` respects the\nbudget, because it asks the Eviction API to evict each Pod; deleting Pods\ndirectly bypasses it:\n\n```yaml\napiVersion: policy/v1\nkind: PodDisruptionBudget\nmetadata:\n    name: booking-pdb\nspec:\n    maxUnavailable: 1\n    selector:\n        matchLabels:\n            app: booking\n```\n\nThe documentation recommends `maxUnavailable` over `minAvailable`, because it\nfollows changes in the number of replicas automatically.\n\nThe database is the hardest tier, because it holds state. PostgreSQL can keep a\nhot standby, which accepts read-only queries, or a warm standby, which starts\naccepting connections only after promotion. Choosing between synchronous and\nasynchronous replication is choosing what a failover may lose. With synchronous replication, commit acknowledgement waits for the configured number of synchronous standbys at the configured acknowledgement level. The settings `synchronous_standby_names` and `synchronous_commit` define those conditions. A failover must choose a standby that has the acknowledged data; an asynchronous standby may still be behind. The price of the first is speed: the documentation\nnotes that slow network links can make a fully synchronous setup lose over half\nits performance.\n\nPostgreSQL does not decide when to fail over. Its documentation says it\nprovides no system software to detect a failed primary and notify the standby;\nthat job, along with moving the IP address, belongs to external cluster\nsoftware. The promotion itself is a single step, `pg_ctl promote` or the\nfunction `pg_promote()`. What follows is the part plans forget. With the old\nprimary gone, the cluster runs on one server, which the documentation calls a\ndegenerate state, until a new standby has been built. The `pg_rewind` utility\ncan speed that up on large clusters.\n\nOn this site, the entry on\n[email with failover and delivery logging](/portfolio/built-email-with-failover-and-delivery-logging/)\nshows the same pattern applied to an outside dependency. Three email\nproviders sit behind one interface, failover between them happens automatically\nrather than as a configuration change made under pressure, and a startup check\nsends a real message, so that staging and production refuse to start without a\nprovider that works. The entry reports that a provider having a bad afternoon\nnow means a degraded path rather than an outage, at the cost of three\nintegrations to keep working. How available a system has to be is a decision of\nits own: the [service levels](/service-levels/) page explains that the\navailability figure is written into the agreement for each system separately,\nsince a report produced overnight and a payment service call for different\nfigures.\n\n## Where it goes wrong\n\n**Redundancy that shares a fate.** Two copies on one host, one power supply,\none switch or one cloud zone fail together. Trace every spare back to what it\ndepends on, and spread replicas across failure domains on purpose rather than\nby luck.\n\n**Split brain.** When cluster nodes lose contact with each other but keep\nrunning, each side may decide the other is dead and start the service itself.\nPacemaker's documentation spells out the result: activating the same IP address\non both hosts makes the destination of incoming packets unpredictable, and for a\ndatabase the damage can be corruption or diverging data. Fencing, also called\nSTONITH (Shoot The Other Node In The Head), is the guard: a fence device, such\nas a power switch the cluster controls, cuts the failed node off before its work\nstarts elsewhere. The documentation states that a cluster cannot safely recover\nfrom an unresponsive node without it, and that the fence device must not rely on\nthe cluster's own network, otherwise losing that network also removes the means\nof fencing.\n\n**A failover nobody has tried.** A standby that has never been promoted is a\nhope, not a plan. Switch roles on a schedule, as PostgreSQL's documentation\nsuggests, write the procedure down, and rehearse it the way chaos engineering\nrehearses other failures.\n\n**Forgetting the degenerate state.** After a failover there is no spare. If\nrebuilding the standby is slow or manual, the next fault, however small,\nbecomes an outage. Treat restoring redundancy as part of the incident, not as a\ntask for later.\n\n**Assuming asynchronous replication loses nothing.** It can lose the last\ntransactions committed before the failure. Decide for each database whether\nthat is acceptable, and make the trade-off visible to the people who own the\ndata.\n\n**Mistaking replication for a backup.** A standby copies a mistaken `DELETE`\nas faithfully as a correct write. High availability keeps the service up; only\na backup lets a team go back to the data as it was.\n\n**Two nodes and no tie-breaker.** Pacemaker's documentation points out that\nlosing a node to a power cut looks the same as losing the network link to it.\nWith two members, each side sees exactly that. Kubernetes asks for at least\nthree control-plane nodes in its stacked layout, and PostgreSQL's documentation\nmentions a third witness server that can prevent some wrong failovers, while\nwarning that the extra complexity is only worth it with careful set-up and\nrigorous testing.\n\n**Paying for nines nobody can see.** Every extra nine costs more than the one\nbefore, and users reach the service through networks and devices less reliable\nthan the target. Choose the number from what users and the business need, then\nbuild to it.\n\n## Further reading\n\n- [ClusterLabs](https://clusterlabs.org/) — the open-source high-availability\n  cluster stack, with its definition of what a cluster coordinates and why it\n  needs fencing.\n- [High Availability, Load Balancing, and Replication](https://www.postgresql.org/docs/current/high-availability.html)\n  — PostgreSQL's chapter on primaries and standbys, and on synchronous and\n  asynchronous designs and their costs.\n- [Failover](https://www.postgresql.org/docs/current/warm-standby-failover.html)\n  — PostgreSQL on heartbeats, witness servers, STONITH, promotion and the\n  degenerate state.\n- [Fencing](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/html/fencing.html)\n  — Pacemaker Explained on split brain, fence devices, and fencing and quorum.\n- [Embracing Risk](https://sre.google/sre-book/embracing-risk/) — the Google\n  SRE book on measuring availability, the cost of each nine and choosing a\n  target.\n- [Disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)\n  — voluntary and involuntary disruptions in Kubernetes, and how a\n  PodDisruptionBudget limits them.\n\n\u003chttps://engineer.company/glossary/high-availability/\u003e\n","date_modified":"2026-11-12T05:34:23Z","description":"What high availability means, how redundancy, failover and fencing keep a service running, what each extra nine costs and where clusters fail.","formats":{"gemtext":"https://engineer.company/glossary/high-availability/index.gmi","gopher":"https://engineer.company/glossary/high-availability/gophermap","html":"https://engineer.company/glossary/high-availability/","json":"https://engineer.company/glossary/high-availability/index.json","markdown":"https://engineer.company/glossary/high-availability/index.md","pdf":"https://engineer.company/glossary/high-availability/index.pdf"},"kind":"page","language":"en","title":"What is high availability?","url":"https://engineer.company/glossary/high-availability/","version":1}