{"canonical_url":"https://engineer.company/glossary/error-budget/","content_markdown":"# Error budget\n\n## What it is\n\nAn error budget is the amount of failure a service is allowed over a stated\nperiod, worked out from its service level objective. If the objective says that\n99.99% of requests must succeed this quarter, the remaining 0.01% is the\nbudget: requests that may fail, by accident or by a risk taken on purpose,\nwithout anyone having broken a promise. It is counted in the same unit as the\nobjective, usually failed requests, and it is spent the moment users meet a\nfailure.\n\nThe idea comes from Google's Site Reliability Engineering book. In its chapter\non embracing risk, product management sets the objective, the monitoring\nsystem acts as a neutral third party that measures what actually happened, and\nthe gap between the two is the unreliability still left for the quarter.\nProduct development and SRE agree that budget together, in advance. The book's\nexample sits at the demanding end of the scale: a service that must answer\n99.999% of queries has a budget of 0.001%, so a single problem that fails\n0.0002% of the quarter's queries has spent a fifth of it.\n\nThe budget does not ask whose fault a failure was. The book is explicit that a\nnetwork outage or a failed datacenter eats into it exactly as a bad release\ndoes, and that the rest of the quarter may then see fewer pushes. Users felt\nthe failure, so it counts. Who has to act on it is a separate question, and the\nanswer belongs in a written policy agreed before anything goes wrong.\n\n## Why it matters\n\nThe people who ship features and the people who keep a service running want\ndifferent things, and without a shared number the argument between them is won\nby whoever negotiates better. The SRE book says such decisions should rest on\nneither politics nor fear nor hope, and recalls Google SRE's unofficial motto:\n\"Hope is not a strategy.\" The budget replaces the argument with a figure both\nsides accepted beforehand. While budget remains, releases go out; when it is\nspent, they pause while the effort goes into testing and resilience instead.\nThe book also describes gentler versions of the same loop, such as slowing\nreleases down or rolling them back as the budget nears empty, and in a footnote\ncalls the plain on-or-off version bang/bang control.\n\nThe more interesting effect is on behaviour. When the budget is large,\ndevelopers can take risks: push more often, try the experiment. When it is\nnearly drained, the book observes, the developers themselves ask for more\ntesting and slower pushes, because nobody wants to stall their own launch. The\nteam becomes self-policing. The book attaches one condition: this works only if\nSRE really has the authority to stop launches once the objective is broken.\n\nReleases are what the budget mostly governs because releases are what mostly\nbreaks things: the SRE Workbook's example policy puts changes at roughly 70% of\nGoogle's outages. That makes the budget a guide to where effort should go. The\nWorkbook's chapter on implementing SLOs works through a service with a 97%\nobjective and a four-week budget of 109,897 failed requests. A bad API release\nthat failed every request for four hours cost 14,066 of them, about 13%. A\ndatabase server that died and took 20 hours to restore from backup cost an\nestimated 72,000, or 65%. The server failure looks far worse, but the example\ncompany sees one of those in five years against two or three bad releases a\nyear, so on average the releases cost about twice as much budget. Fixing the\nrelease process pays off first.\n\nA budget that is never spent says something too. The book notes that it\nexposes what an overly high target costs in inflexibility and slow innovation,\nand that a team struggling to launch can choose to loosen its objective, which\nenlarges the budget.\n\n## What it looks like in practice\n\nSpending is a speed as well as a total. The Workbook's chapter on alerting\ncalls that speed the burn rate: how fast, relative to the objective, the\nservice is consuming its budget. At a burn rate of 1 the budget runs out\nexactly at the end of the window. For a 99.9% objective over 30 days, its table\nworks out like this:\n\n```text\nBurn rate   Error rate   Budget gone after\n1           0.1%         30 days\n2           0.2%         15 days\n10          1%           3 days\n1,000       100%         43 minutes\n```\n\nAlerting on burn rate beats paging whenever errors pass 0.1%, which would wake\nsomeone for every brief spike. The Workbook recommends checking several burn\nrates at once, each over a long window and a short one. Its starting point for\na 99.9% objective pages when 2% of the 30-day budget goes in one hour (a burn\nrate of 14.4) or 5% goes in six hours (a burn rate of 6), and opens a ticket\nwhen 10% goes in three days (a burn rate of 1). The short window, a twelfth of\nthe long one, makes the alert stop soon after the problem does. In the\nWorkbook's example, the errors end and five minutes later the short-window\naverage is back under the threshold, while the one-hour average stays above it\nfor a full hour.\n\nAs a Prometheus rule file, the two paging conditions read as follows. The\nratios are recording rules named the way the Workbook names them, each holding\nthe share of failed requests over the window in its name:\n\n```yaml\ngroups:\n  - name: orders-error-budget\n    rules:\n      - alert: OrdersErrorBudgetBurn\n        expr: |\n          (\n            job:slo_errors_per_request:ratio_rate1h{job=\"orders\"}\n              \u003e (14.4 * 0.001)\n            and\n            job:slo_errors_per_request:ratio_rate5m{job=\"orders\"}\n              \u003e (14.4 * 0.001)\n          )\n          or\n          (\n            job:slo_errors_per_request:ratio_rate6h{job=\"orders\"}\n              \u003e (6 * 0.001)\n            and\n            job:slo_errors_per_request:ratio_rate30m{job=\"orders\"}\n              \u003e (6 * 0.001)\n          )\n        labels:\n          severity: page\n```\n\nFollow one bad deploy through it. At 14:00 a release starts failing 2% of\norder requests, a burn rate of 20. The five-minute ratio passes the 1.44% line\nwithin four minutes, but the one-hour ratio reaches it only after the errors\nhave run for about 43 minutes, so the page goes out at around 14:43. By then\nexactly 2% of the month's budget is gone, which is the trade the rule makes: a\nshort blip wakes nobody, and a sustained burn cannot hide. Roll back at 14:50\nand the five-minute ratio falls below the line within two minutes, ending the\npage; the one-hour ratio alone would have kept it firing until about 15:07.\n\nKeeping paging proportionate is the same concern behind the company's\n[first-party error monitoring and tracing](/portfolio/built-first-party-error-monitoring-and-tracing/),\nfrom 2025. It detects spikes and regressions with a cooldown, so that one bad\ndeploy does not page anyone forty times, its alerting rules were fitted to that\nplatform rather than to a generic one, and one of its 11 operator views is given\nto service level objectives.\n\n## Where it goes wrong\n\n**Counting a failure is confused with blaming someone for it.** Everything\nusers feel spends the budget, but not everything should stop the team's own\nreleases. The Workbook's example policy lets feature work continue when a\ncompany-wide network problem caused the outage, or when another team's service\ndid and that team has frozen its own releases; when the team's own bug or\nprocedure spent the budget, reliability comes first. For an outage caused by\nanother team's dependency, the Implementing SLOs chapter sets out two schools\nof thought, carry on or freeze anyway, and says the freeze makes users happier.\nChoose in advance, write the choice into the policy, and name who settles a\ndisagreement about the arithmetic; the example policy sends it to the CTO.\n\n**Errors land in the wrong category.** A failed load test or a penetration\ntester's rejected requests spend budget although no user was affected, and the\nexample policy names miscategorised errors as a way budget disappears for\nnothing. The reverse is worse: real failures filed as something harmless keep\nthe service inside its objective on paper while users suffer, and the same\npolicy makes reliability work mandatory when that is found. Keep traffic the\nobjective does not cover out of the indicator, and check the classification\nafter any incident whose cost looked wrong.\n\n**A large incident is paid for and forgotten.** A freeze buys time, not\nunderstanding. The example policy ties the two together: a single incident\nthat spends more than 20% of the four-week budget requires a postmortem with at\nleast one action item of the highest priority, and one class of outage that\nspends more than 20% in a quarter must appear on the next quarter's plan.\n\n**The alert can never fire.** Thresholds copied from the 99.9% table do not\ncarry over to other targets. At 90% availability, the Workbook points out, a\ntotal outage spends only 1.4% of the budget in an hour, so a rule that pages\nat 2% an hour stays silent through it. At the other extreme, a 99.999% monthly\ntarget is exhausted by a total outage in 26 seconds, faster than many\nmonitoring systems even collect a sample. Tune the windows and thresholds to\nthe target, and for very high targets accept the Workbook's conclusion that\nonly design defends them: roll a change out to 1% of users first, and the same\nfailure leaves 43 minutes before the budget is gone.\n\n**Low traffic turns one failure into a page.** At ten requests an hour, a\nsingle failed request is a 10% error rate for that hour. Against a 99.9%\nobjective, the Workbook notes, that one request pages at once, because it\nspends 13.9% of the 30-day budget: the whole month allows only seven failures.\nThe Workbook's remedies are to generate artificial traffic, to combine several\nsmall services into one for monitoring, or to change the product so that one\nfailed request matters less.\n\n**The target is raised before the service can meet it.** A team that needs a\ntighter objective and adopts it straight away lives in permanent emergency,\nwith the budget always spent. The Workbook suggests an aspirational objective\ninstead: measured and tracked beside the current one, but named in the policy\nas requiring no action until the service is ready for it.\n\n## Further reading\n\n- [Embracing Risk](https://sre.google/sre-book/embracing-risk/) — chapter 3 of\n  Google's Site Reliability Engineering book, where the error budget is\n  introduced as a way to settle release decisions with data.\n- [Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/) — the SRE\n  Workbook on burn rates, multiwindow alerts and low-traffic services.\n- [Example Error Budget Policy](https://sre.google/workbook/error-budget-policy/)\n  — a complete policy: when releases stop, which outages are exempt, and when a\n  postmortem is required.\n- [Implementing SLOs](https://sre.google/workbook/implementing-slos/) — how to\n  compute a budget from an objective and use it to choose between reliability\n  investments.\n- [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)\n  — the Prometheus documentation for the rule file format used above.\n\n\u003chttps://engineer.company/glossary/error-budget/\u003e\n","date_modified":"2026-10-29T11:17:41Z","description":"How an error budget is counted and spent, what burn rate means and how to page on it, and what a team agrees to do when the budget runs out.","formats":{"gemtext":"https://engineer.company/glossary/error-budget/index.gmi","gopher":"https://engineer.company/glossary/error-budget/gophermap","html":"https://engineer.company/glossary/error-budget/","json":"https://engineer.company/glossary/error-budget/index.json","markdown":"https://engineer.company/glossary/error-budget/index.md","pdf":"https://engineer.company/glossary/error-budget/index.pdf"},"kind":"page","language":"en","title":"What is an error budget?","url":"https://engineer.company/glossary/error-budget/","version":1}