{"canonical_url":"https://engineer.company/glossary/canary-release/","content_markdown":"# Canary release\n\n## What it is\n\nA canary release puts a new version of a service in front of a small share of\nits real traffic first, while the version it replaces keeps serving everyone\nelse. If the newcomer holds up, the share grows in steps until it carries\neverything and the old version is retired. If it does not, the traffic goes back\nto the old version, which never stopped running, and only the small share ever\nsaw the fault. Danilo Sato described the technique on Martin Fowler's bliki in\nJune 2014 and noted that it also goes by phased rollout or incremental rollout.\nKubernetes tooling tends to say canary deployment: Argo Rollouts, a Kubernetes\ncontroller that adds release strategies of this kind, names the setting\n`canary`.\n\nThe name comes from coal mining. Miners took a canary below ground in a cage,\nbecause leaking gas would overcome the bird before it reached the people, and\nthe bird's distress was their warning to get out. The small share of traffic\nplays the same part: it meets the trouble first, while there is still time to\nact on it.\n\nWho lands in that share is a choice. Sato lists a random sample of users, a\ncompany's own staff before the public, and users picked by profile or\ndemographics. Large systems often split along lines they already have instead:\none region, one location or one brand first. Facebook, in Sato's account, used\nseveral canaries, the first visible only to its own employees and with every\nfeature flag turned on, so that problems with new features showed up early.\n\nThe contrast with blue-green deployment is the shape of the move. Blue-green\nbuilds a complete second copy and moves all traffic in one switch, keeping the\nold copy as the way back. A canary moves traffic gradually, so for a while two\nversions serve real users side by side. A rolling update also has two versions\nlive at once, but it replaces instances as fast as they become ready, with no\npause to judge how the new version is doing. A canary stops at each step and\nwaits for an answer.\n\n## Why it matters\n\nSome faults only show up in production: under real load, with real data, behind\nthe real mix of clients. A staging environment can come close but is never\nidentical. A canary accepts that and limits the damage instead. If one request\nin twenty reaches the new version, a bad release hurts one request in twenty for\nas long as it takes to notice, and the fix is to send that share back.\n\nSato points to a second use: capacity testing. Ramping the share up slowly shows\nhow the new version behaves under growing production load, with a safe way back,\ninstead of building a separate environment just for load tests, which can never\nbe quite as production-like as production itself.\n\nThe catch is that a canary is only as good as the signal that judges it. Errors,\nlatency and the business numbers that matter, such as sign-ups, payments or\norders, have to be visible per version and early enough to stop the next step.\nWithout that, a canary is a slow rollout with extra waiting. Sato notes that\nIMVU pioneered what is known as a \"cluster immune system\": watching business\nmetrics and rolling a release back automatically on a statistically significant\nregression.\n\nThere are running costs too. Two or more versions are in service at once, and\neverything they share, the database, a cache, a queue, the API that clients\ncall, has to work with both for the duration. Sato advises keeping the number of\nversions live at the same time to a minimum. The technique also fits poorly with\nsoftware installed on users' own computers or phones, where the vendor does not\ndecide when an upgrade happens. There, he suggests keeping the backend able to\nserve both versions, watching which client versions are still in use, and\ndropping the old one only when its share has fallen far enough.\n\n## What it looks like in practice\n\nAt its plainest, a canary on Kubernetes is two Services, one selecting the\nstable pods and one selecting the canary pods, and a route that weights traffic\nbetween them. With the Gateway API, the weights live in an HTTPRoute:\n\n```yaml\napiVersion: gateway.networking.k8s.io/v1\nkind: HTTPRoute\nmetadata:\n  name: web\nspec:\n  parentRefs:\n    - name: prod-web\n  hostnames:\n    - \"www.example.com\"\n  rules:\n    - backendRefs:\n        - name: web-stable\n          port: 8080\n          weight: 95\n        - name: web-canary\n          port: 8080\n          weight: 5\n```\n\nWeights are proportions, not percentages: each backend gets its weight divided\nby the sum of the weights in that rule, so 95 and 5 send one request in twenty\nto the canary. An omitted weight counts as 1, and a weight of 0 sends nothing.\nThe Gateway API guide to traffic splitting also shows a rule that matches a\nrequest header, `traffic: test` in its example, and sends only those requests to\nthe new version, so testers can reach it before any customer does. Moving the\ncanary forward is editing two numbers; pulling it back is setting its weight\nto 0.\n\nArgo Rollouts automates the steps. A `Rollout` takes the place of a\n`Deployment`, with the same pod template, and lists the steps the controller\nwalks through whenever that template changes:\n\n```yaml\napiVersion: argoproj.io/v1alpha1\nkind: Rollout\nmetadata:\n  name: web\nspec:\n  replicas: 10\n  selector:\n    matchLabels:\n      app: web\n  template:\n    metadata:\n      labels:\n        app: web\n    spec:\n      containers:\n        - name: web\n          image: registry.example.com/web:1.4.2\n  strategy:\n    canary:\n      maxSurge: \"25%\"\n      maxUnavailable: 0\n      steps:\n        - setWeight: 10\n        - pause:\n            duration: 1h\n        - setWeight: 20\n        - pause: {}\n```\n\nWithout a traffic router, Argo approximates the weight with pod counts: ten\nreplicas at 10 per cent means one canary pod and nine stable ones. The first\npause waits an hour; the second waits until someone decides. A release and its\ntwo possible endings look like this:\n\n```sh\nkubectl argo rollouts set image web web=registry.example.com/web:1.5.0\n# at the open-ended pause, either carry on ...\nkubectl argo rollouts promote web\n# ... or stop and put the stable version back in front of everyone\nkubectl argo rollouts abort web\n```\n\nEverything in that sequence hinges on knowing, during the pause, whether the\ncanary is doing worse than what it replaced. That is the work behind the\n[Site Reliability \u0026 Monitoring](/services/site-reliability/) service, which is\nbuilt around alerts that speak up while a fault is still small. The\n[first-party error monitoring and tracing](/portfolio/built-first-party-error-monitoring-and-tracing/),\nfrom 2025, shows what such a signal looks like: errors are grouped into\nrecurring problems rather than listed one by one, and spikes and regressions are\ndetected with a cooldown, so that one bad deploy does not page anyone forty\ntimes. A canary step needs exactly that: a regression that makes itself known\ninside the pause, not one a user stumbles on the next morning.\n\n## Where it goes wrong\n\n**The canary is too small to say anything.** One pod in ten at a quiet hour may\nserve a few hundred requests during its pause, and a fault that hits one request\nin a thousand never shows. The clock is the wrong measure; the number of\nrequests the canary has served is the right one. Size the share and the pause\ntogether, so that a rare error has the chance to appear several times, and hold\na step longer at night than at noon.\n\n**The dashboards cannot tell canary from stable.** A canary that fails one\nrequest in fifty more than the stable version does, at a 5 per cent share, adds\nabout one failed request in a thousand to the total. On a dashboard with one\nerror rate for the whole service, that is invisible. Give canary and stable pods\ndifferent labels, which Argo Rollouts adds through `canaryMetadata` and\n`stableMetadata`, carry the label into the metrics, and compare the canary with\nthe stable version rather than with a fixed threshold. A background `analysis`\non the Rollout aborts the update when it fails.\n\n**Both versions write to the same data.** For the length of the rollout, old and\nnew code read and write the same database, and with random routing a single\nuser's requests may land on both within a minute. A new column the old version\nnever fills, or a format it cannot read, turns into errors on the stable side,\nthe part that was supposed to be safe. Sato's answer is parallel change: the\ndatabase supports both versions of the application for the whole rollout, and\nthe old shape goes only once the old version is gone.\n\n**An abort stops the traffic, not the change.** `kubectl argo rollouts abort`\nmoves traffic back to the stable ReplicaSet, but the Rollout's spec still names\nthe new image, and resuming the Rollout makes that image the target once more.\nThe rollback is complete only when the template is set back to the previous\nversion. If the spec is kept in a repository, make the change there, so that the\nsource and the cluster agree on what should be running.\n\n**The canary doubles as an A/B test.** The routing looks the same, so it is\ntempting to measure a new feature's effect on conversions with the canary's\ntraffic. Sato advises keeping the two apart: a canary hunts for problems and\nshould finish within minutes or hours, while an experiment tests a hypothesis\nand may need days of data to be statistically significant. If the canary is\njudged on business metrics, the effect the experiment is meant to cause also\nmuddies the verdict. Release with the canary, then run the experiment behind its\nown split, such as a feature flag, once the version is out.\n\n## Further reading\n\n- [Canary Release](https://martinfowler.com/bliki/CanaryRelease.html)\n  — Danilo Sato's 2014 description on Martin Fowler's bliki: choosing users,\n  capacity testing, A/B tests and the database.\n- [Argo Rollouts: Canary](https://argo-rollouts.readthedocs.io/en/stable/features/canary/)\n  — steps and pauses, weights approximated by pod counts, and scaling with a\n  traffic router.\n- [HTTP traffic splitting](https://gateway-api.sigs.k8s.io/guides/user-guides/traffic-splitting/)\n  — the Gateway API guide to weighted backends and a header-matched canary\n  route.\n- [kubectl argo rollouts abort](https://argo-rollouts.readthedocs.io/en/stable/generated/kubectl-argo-rollouts/kubectl-argo-rollouts_abort/)\n  — what an abort does to traffic, and what it leaves in the spec.\n\n\u003chttps://engineer.company/glossary/canary-release/\u003e\n","date_modified":"2026-10-30T02:51:31Z","description":"Sending a new version to a small share of real traffic first: how the share is routed, what judges it, and where a small share stops telling the truth.","formats":{"gemtext":"https://engineer.company/glossary/canary-release/index.gmi","gopher":"https://engineer.company/glossary/canary-release/gophermap","html":"https://engineer.company/glossary/canary-release/","json":"https://engineer.company/glossary/canary-release/index.json","markdown":"https://engineer.company/glossary/canary-release/index.md","pdf":"https://engineer.company/glossary/canary-release/index.pdf"},"kind":"page","language":"en","title":"What is a canary release?","url":"https://engineer.company/glossary/canary-release/","version":1}