{"canonical_url":"https://engineer.company/glossary/chaos-engineering/","content_markdown":"# Chaos engineering\n\n## What it is\n\nChaos engineering is the practice of injecting real-world failures into a\nrunning system on purpose, under control, to find out whether it copes. The\nPrinciples of Chaos Engineering, the short document most practitioners point\nto, defines it as \"the discipline of experimenting on a system in order to\nbuild confidence\" that the system can withstand turbulent conditions in\nproduction. The word that matters is experimenting. A chaos experiment does not\ncheck a feature against a specification; it asks a question about the whole\nsystem and accepts whatever answer comes back.\n\nThe case for it starts with distributed systems. Every service in one can be\nworking correctly and the interactions between them can still produce outcomes\nnobody predicted, and rare real-world events such as a crashed server or a\nsevered network link make that worse. The Principles list the kinds of weakness\nthat hide there: fallback settings that are wrong when a service is unavailable,\npoor timeout settings that amplify retries, dependencies overwhelmed by their\ncallers, and one failed component taking several others down. None of those\nshows up when each part is tested on its own.\n\nAn experiment has four steps in the Principles' version:\n\n1. Define steady state as a measurable output of the system that indicates\n   normal behaviour, such as throughput, error rate or a latency percentile.\n2. Hypothesise that the steady state will continue in both a control group and\n   an experimental group.\n3. Introduce variables that reflect real events: servers that crash, disks that\n   fail, network connections that are cut.\n4. Compare the groups' measured outputs and look for evidence that contradicts\n   the prediction.\n\nThe harder the steady state is to disturb, the more confidence the team can\nhave in the system. When the hypothesis falls, the team has found a weakness\nwhile it is still small and while people are watching, rather than during an\noutage at night.\n\nThe best-known tool is Netflix's Chaos Monkey, which randomly terminates virtual\nmachine instances and containers in production. Its README calls it an example\nof a tool that follows the Principles, and its stated purpose is to make\nengineers build services that are resilient to instance failures.\n\n## Why it matters\n\nRedundancy, failover and fallbacks are all promises about what happens when\nsomething breaks, and a promise that has never been tested is a hypothesis. A\nstandby that has never taken traffic, a retry policy that has never met a slow\ndependency, an alert that has never fired: each of these may work, and the\nfirst real failure is an expensive way to find out. Chaos engineering moves\nthat discovery to a time the team chooses.\n\nIt also tests the system rather than the parts. Unit and integration tests\ncheck behaviour someone already thought of. The Principles make the\ndistinction directly: chaos experiments verify that the system does work,\ninstead of trying to validate how it works. That is why they focus on the\noutput a user would notice, such as error rate and latency, and not on the\ninternals of any one component.\n\nProduction is where the evidence is. Systems behave differently depending on\ntheir environment and traffic patterns, and the Principles argue that sampling\nreal traffic is the only reliable way to capture the request path. Google's Site\nReliability Engineering book reaches the same conclusion from the other\ndirection: synthetic load may not show where components really break, so once a\nservice has protection against overload, it suggests exercising failures against\na limited part of the live service, with spare capacity ready in case the\nprotections do not work and someone has to fail over by hand.\n\nFinally, it changes incentives. Chaos Monkey's README puts the idea plainly:\nexposing engineers to failures more often gives them a reason to build\nresilient services. A team whose instances can be terminated on any working\nday designs for that from the start, instead of discovering the gap on the day\na machine dies without warning.\n\n## What it looks like in practice\n\nA chaos experiment is written down before it is run, much like a test plan.\nHere is one for an invented online shop, with every number made up for the\nexample:\n\n```text\nExperiment:   the payment provider slows down\nSteady state: checkout success rate and p99 checkout latency, per 5 min\nHypothesis:   with 300 ms added to calls to the payment provider, the\n              experimental group's success rate stays within 0.2 points\n              of the control group's\nGroups:       2% of checkout traffic in each, assigned at random\nVariable:     300 ms of latency on the payment client, nothing else\nAbort if:     success rate gap above 1 point for 2 minutes in a row\nWindow:       30 min, working hours, on-call engineer told in advance\nResult:       hypothesis disproved; retries tripled the calls to the\n              provider and p99 latency passed the 1 s timeout\nFollow-up:    a retry budget per request; rerun the experiment after\n```\n\nA disproved hypothesis is the useful outcome here. The weakness is found on 2%\nof traffic for a few minutes, with an abort condition already agreed, and the\nfollow-up is an engineering change rather than an apology.\n\nThe variable itself is often injected with ordinary tools. On Linux, the netem\nqueueing discipline emulates the properties of real networks, including delay,\nloss, duplication and corruption. For example:\n\n```sh\n# experiment 1: add 100 ms of delay to every packet leaving eth0\ntc qdisc add dev eth0 root netem delay 100ms\ntc qdisc del dev eth0 root\n\n# experiment 2: drop one packet in a thousand, at random\ntc qdisc add dev eth0 root netem loss 0.1%\ntc qdisc del dev eth0 root\n```\n\nThe `del` line is the important one, and it belongs in a cleanup step that\nruns whether the experiment passed, failed or crashed halfway. Netem also takes\na `seed` so that random loss can be reproduced, and its manual warns that for\nrealistic TCP performance results netem has to be placed on the ingress of the\nreceiving host.\n\nAt the level of whole machines, Chaos Monkey shows what careful defaults look\nlike. Out of the box `enabled` is false and `leashed` is true, which means\nterminations are only simulated and logged. When it is switched on, it works through groups of\ninstances (by app, stack or cluster), with a mean time between terminations of\ntwo days per group and a minimum of one day, on work days only, between 09:00\nand 15:00 in its configured time zone. Accounts, regions and stacks can be\nexcluded, and its configuration has a slot for an outage checker that tells it\nwhether an outage is under way; the shipped implementations of that slot do\nnothing, so the check is the operator's to supply. The SRE book's own list of\nproduction failure tests is just as concrete: reduce task counts quickly or\nslowly, lose a cluster's worth of capacity at once, and blackhole backends,\nincluding the noncritical ones whose silence can still stall a front end.\n\nOn this site, the case study of how we\n[proved the intrusion ban works every run](/portfolio/proved-the-intrusion-ban-works-every-run/),\nfrom September 2026, is not a chaos experiment in the full sense, but it has\nthe same shape. Every hardening run bans an address from the block the\nstandards reserve for documentation, reads the resulting rule back from the\nkernel rather than trusting the service's own status, and unbans it in a\ncleanup block that runs whether the check passed or failed. Because the test\naddress belongs to nobody, a stray ban that outlives a failed run cannot lock\nout a real network, and the page is open about the cost: writing a rule into\nproduction on every run is a real intrusion into a live system, accepted\nbecause the alternative had already been shown to be worse. The\n[backup and restore-test implementation](/portfolio/backups-that-have-actually-been-restored/)\napplies that idea to recovery in its design: a monthly workflow restores a known file to a\nscratch location and compares its bytes. Its implementation record dated 2026-10-27 says the\ntimers were disabled without a configured repository and records no completed live restore.\nThe workflow exists in code; a successful recovery exercise remains a separate deliverable.\n\n## Where it goes wrong\n\n**Breaking things without a hypothesis.** Killing a random server and watching\nwhat happens is a demonstration, not an experiment. Write down the steady state,\nthe hypothesis and the abort condition first. If the team cannot see its steady\nstate on a dashboard in real time, that is the first thing to fix, and the\nexperiment can wait.\n\n**Watching internals instead of output.** CPU on one host or the length of one\nqueue says little about whether users noticed. The Principles ask for\nmeasurable output of the whole system, such as throughput, error rates and\nlatency percentiles, because that is what the hypothesis is about.\n\n**An unbounded blast radius.** The Principles accept some short-term negative\nimpact but make containing the fallout the engineer's responsibility. Start\nwith one instance or a small percentage of traffic, keep an abort switch that\nworks without the tool that caused the damage, and never run an experiment on\ntop of a live incident.\n\n**Never leaving staging.** A staging environment is a sensible first step and a\npoor last one, because its traffic, data and neighbours are not production's.\nMove to a small slice of real traffic once the experiment has run cleanly\nbefore production, rather than treating staging results as the answer.\n\n**Running it once.** A system that survived an experiment in March has been\nchanged many times since. The Principles call manual experiments\nlabour-intensive and ultimately unsustainable, and ask for experiments\nautomated to run continuously, so that a regression in resilience shows up the\nweek it is introduced.\n\n**Only ever killing machines.** Clean crashes are the easy case, because\nsomething notices them. The Principles also name software failures such as\nmalformed responses and non-failure events such as a traffic spike or a scaling\nevent, and the SRE book adds a backend that accepts requests and never\nanswers, which can still hurt a front end when requests have long deadlines.\nPrioritise variables by likely impact or by how often they happen.\n\n**Findings that go nowhere.** A disproved hypothesis is a weakness with a name.\nGive it an owner and a priority in the same tracker as everything else, and\nrerun the experiment once the fix ships to confirm that the steady state now\nholds.\n\n## Further reading\n\n- [Principles of Chaos Engineering](https://principlesofchaos.org/) — the\n  definition, the four steps of an experiment and the five advanced principles,\n  last updated in March 2019.\n- [Chaos Monkey](https://netflix.github.io/chaosmonkey/) — Netflix's\n  documentation for its instance-termination tool, which requires Spinnaker.\n- [Chaos Monkey: configuring behavior via Spinnaker](https://netflix.github.io/chaosmonkey/Configuring-behavior-via-Spinnaker/)\n  — termination frequency, grouping and exceptions, with the default values.\n- [Chaos Monkey: configuration file format](https://netflix.github.io/chaosmonkey/Configuration-file-format/)\n  — every setting and its default, including the leashed mode and the\n  working-hours window.\n- [Addressing Cascading Failures](https://sre.google/sre-book/addressing-cascading-failures/)\n  — chapter 22 of Google's Site Reliability Engineering book, including testing\n  until failure and production failure tests.\n- [tc-netem(8)](https://man7.org/linux/man-pages/man8/tc-netem.8.html) — the\n  Linux manual page for the network emulator used to inject delay, loss and\n  corruption.\n\n\u003chttps://engineer.company/glossary/chaos-engineering/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"What chaos engineering is, how an experiment against steady state works, what tools such as Chaos Monkey and netem do, and where the practice goes wrong.","formats":{"gemtext":"https://engineer.company/glossary/chaos-engineering/index.gmi","gopher":"https://engineer.company/glossary/chaos-engineering/gophermap","html":"https://engineer.company/glossary/chaos-engineering/","json":"https://engineer.company/glossary/chaos-engineering/index.json","markdown":"https://engineer.company/glossary/chaos-engineering/index.md","pdf":"https://engineer.company/glossary/chaos-engineering/index.pdf"},"kind":"page","language":"en","title":"What is chaos engineering?","url":"https://engineer.company/glossary/chaos-engineering/","version":1}