Blue-green deployment

What it is

Blue-green deployment is a way of releasing software with two
production environments built to be as alike as possible. One of
them, called blue here, serves every request. The new version is
installed on the other, green, and tested there while no customer
traffic reaches it. When it is ready, whatever sits in front of the
two is changed so that every new request goes to green. Blue keeps
running untouched, and if green misbehaves, the same change made in
reverse puts blue back in service.

The colours carry no meaning, which is the point of them. Names such
as "old" and "new" or "primary" and "standby" swap meaning after
every release, while two colours simply take turns: after the next
switch green is live and blue is the idle one. Martin Fowler
described the technique on his bliki in 2010 and attributes the
name, a little hazily, to Daniel Terhorst-North and Jez Humble. The
usual spelling has a hyphen, though tools write it their own way:
Argo Rollouts, a Kubernetes controller that adds deployment
strategies of this kind, calls it `blueGreen` in its configuration.

The switch can be any single point that decides where requests go.
Fowler's version changes a router. It can equally be a load
balancer's target, a reverse proxy's upstream, or in Kubernetes a
Service whose label selector picks the Pods that receive its
traffic. The two environments do not have to be two data centres
either: Fowler lists separate hardware, separate virtual machines,
or two zones on one system with their own IP addresses as all good
enough.

Two neighbouring techniques make the contrast clear. A rolling
update replaces instances a few at a time, so old and new versions
answer requests side by side until the last old one is gone, and
rolling back means rolling forward again in the other direction. A
canary release sends a small share of traffic to the new version
first and widens it step by step. Blue-green moves everything at
once, but keeps a complete, running copy of the previous version, so
the way back is one step, the same size as the way forward.

Why it matters

A release becomes a short, reversible event instead of a long,
one-way one. The slow parts, copying files, starting processes,
warming caches, happen on green while blue is still serving, so
customers never wait on them. The switch itself is quick, and so is
undoing it: nothing is rebuilt or reinstalled, because the previous
version never stopped. That changes how a team feels about releasing
on an ordinary afternoon, and how quickly it can act when something
looks wrong.

Green can also be checked in conditions very close to production
before any customer sees it: the same network, the same
configuration, the same database. A smoke test against green proves
far more than the same test in a staging environment that differs in
a dozen small ways. Argo Rollouts formalises this with a preview
Service, which routes to the new version before promotion so it can
be tested without serving production traffic.

The idle environment has a second job. Once a release has settled,
the side that just went quiet becomes the place to stage the next
one, and in the meantime it is a warm standby. Fowler points out
that the same mechanism "allows you to test your disaster-recovery
procedure on every release", because every switch is a failover that
someone is watching.

The costs are plain. Two production environments need two
environments' worth of capacity, at least for the duration of a
release. And while the application tier is easy to duplicate, the
data is not: blue and green nearly always share one database, which
means every release has to be readable by both versions. A team that
cannot guarantee that gets a fast switch forward and a broken switch
back.

What it looks like in practice

On Kubernetes, the smallest working version is two Deployments and
one Service. The Deployments `web-blue` and `web-green` run the same
application with the labels `app: web` and `slot: blue` or `slot:
green`. The Service that customers reach selects one slot:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
    slot: blue
  ports:
    - port: 80
      targetPort: 8080
```

A release puts the new image on the idle side, waits for it, tests
it through a second Service that selects `slot: green` alone, and
then moves the live one:

```sh
kubectl set image deployment/web-green web=registry.example.com/web:1.5.0
kubectl rollout status deployment/web-green --timeout=5m
i# smoke tests against the web-green preview Service go here		engineer.company	70
kubectl patch service web -p '{"spec":{"selector":{"slot":"green"}}}'
i# the way back is the same command with the other colour		engineer.company	70
kubectl patch service web -p '{"spec":{"selector":{"slot":"blue"}}}'
```

`kubectl patch` defaults to a strategic merge patch, so only the
`slot` key of the selector changes and `app: web` stays where it
was. The Service controller keeps scanning for Pods that match the
selector and updates the Service's endpoints when the set changes,
so new connections land on green while blue sits ready with the
previous image.

Argo Rollouts automates the same dance with one `Rollout` object: an
`activeService` that it points at the new pods on promotion, and an
optional `previewService` that reaches them before. With
`autoPromotionEnabled: false` the rollout pauses before the switch
until someone runs `kubectl argo rollouts promote web`.

The DevOps and CI/CD service promises changes that are tested,
released and reversible, and blue-green deployment is one way to
make the last of those cheap. It rests on two habits that
deployments of one platform to Azure show at work, without any
second production slot: Bicep templates keep the target
infrastructure identical from one environment to the next, and the
same built artefact is promoted from development through staging to
production, picking up each environment's settings where it lands.
That route moves a release through environments in a line.
Blue-green adds a twin at the end of it, and the same two habits,
identical infrastructure and one unchanged artefact, are what let a
team trust that the twin really is one.

Where it goes wrong

A schema change turns the way back into a dead end. Green starts,
runs a migration that renames a column, and serves traffic happily.
Ten minutes later a fault sends traffic back to blue, which still
asks for the old column name and fails on every request. The
rollback was planned for the application and not for the data.
Fowler's advice from 2015 is to separate the two: ship a database
change that both versions can work with, then the application, and
remove what only the old version needed once the new one has
settled.

State written on green is stranded by a switch back. Anything green
keeps for itself, such as files uploaded to its local disk, an
in-memory session store or its own cache of pending jobs, is not on
blue when traffic returns there. Users find themselves logged out or
missing what they just saved. Keep shared state in stores both
colours use. Where green must have its own data, Fowler suggests
feeding transactions to both environments, or starting green
read-only and switching it to read-write only once it has proved
itself.

The way back is scaled away before the fault shows. Argo Rollouts
scales the old ReplicaSet down after the switch, and
`scaleDownDelaySeconds` defaults to

30. A fault that appears after a quarter of an hour, a nightly job
or a slow leak, finds no blue to return to. Set the delay to cover
the time it usually takes for trouble to surface, and accept that
the price is the capacity:

```yaml
strategy:
  blueGreen:
    activeService: web
    previewService: web-preview
    autoPromotionEnabled: false
    scaleDownDelaySeconds: 3600
```

Promotion follows readiness, not correctness. `autoPromotionEnabled`
defaults to true, which means Argo Rollouts switches as soon as the
new pods are ready. A readiness probe tells you a process answers,
not that checkout works. Pause before promotion as in the block
above and test through the preview Service, or attach a
`prePromotionAnalysis`, which aborts the rollout when it fails, and
a `postPromotionAnalysis`, which switches back to the previous
version when it fails after the switch.

The idle side drifts. Between releases, someone applies an urgent
configuration change to the live environment only. The next release
lands on the idle side, which still has the old setting, and the
switch that was supposed to change one thing changes two. Build both
slots from the same templates and the same configuration source, so
a change to one is a change to both, and treat the idle side as the
next release's staging area, as Fowler describes.

Further reading

  * Blue Green Deployment — Martin Fowler's 2010 description, with
his 2015 note on database changes.

  * Argo Rollouts: BlueGreen — active and preview Services,
automatic promotion, analysis before and after the switch, and the
scale-down delay.

  * Kubernetes Service — how a selector picks the Pods that receive
a Service's traffic.

  * kubectl patch — strategic merge, merge and JSON patches, the
command that flips the selector in the example above.

hThis page on the web	URL:https://engineer.company/glossary/blue-green-deployment/	engineer.company	70
