{"canonical_url":"https://engineer.company/glossary/immutable-infrastructure/","content_markdown":"# immutable infrastructure\n\n## What it is\n\nImmutable infrastructure is a way of running servers in which a machine, once it\nis in service, is never changed. HashiCorp's Well-Architected Framework\ndescribes it as infrastructure that, “once deployed, is never modified—only\nreplaced”. A new version of the application, an upgraded library, a security\npatch and a changed setting all arrive the same way: as a new machine or\ncontainer, built from a new image, which takes the place of the old one. The old\none is then switched off and deleted.\n\nThe opposite is mutable infrastructure, where a long-lived server is updated\nwhere it stands. In June 2013 Chad Fowler described what such a server turns\ninto after years of upgrades: patched over and over, carrying configuration\nedits made by hand in the middle of an outage, running cron jobs that only one\nperson knows about, until the team is afraid both to change it and to replace\nit. His remedy, which he called immutable deployments, was to build every system\nthrough automation and leave it untouched from then on. An upgrade or a new\nrelease then means building a fresh server or image and throwing the old one\naway.\n\nTwo kinds of image make the approach practical. A machine image, in the words of\nPacker's documentation, is a single static unit that holds a pre-configured\noperating system and the installed software. Packer, a community tool, builds\nthem: from one source configuration it produces matching images for several\nplatforms. Container images are the second kind, and Docker's documentation is\nplain about them: once an image has been created it cannot be modified, and the\nonly options are to make a new image or to add changes on top of it as new\nlayers.\n\nThe term describes how a system is operated, not a product you install. It also\nhas a boundary. The application servers can be replaced on every release, but\nthe data they work on has to live somewhere that outlasts each replacement, and\nthat boundary is where most of the practical design work goes.\n\n## Why it matters\n\nThe first reason is drift. HashiCorp's framework names it directly: manual\nupdates and in-place modifications make servers diverge from their original\nconfiguration over time. Two servers that started from the same setup a year\napart need not end up identical, because each received a different sequence of\nfixes, retries and hand edits. When every server is a fresh copy of one image,\nthe image is the complete description of the server, and there is nothing left\nto drift.\n\nThe second reason is rollback. If a release misbehaves, the framework's answer\nis to deploy the previous artifact again rather than repair the servers that\nare running. The previous image is a known state that has already run in\nproduction, so the team does not have to reconstruct what the server looked like\nyesterday, under pressure, from memory and shell history.\n\nThe third reason is fewer hands on production. The framework points out that\nconnecting to production servers to apply updates by hand introduces human\nerrors such as typos and forgotten steps, and that with immutable infrastructure\nnobody has to log in to a server by hand at all. Fewer people need a login on a\nlive machine at all, and the ones who have one rarely need to use it.\n\nThe fourth reason is the audit trail. Auditors, the same page notes, want proof\nthat production has not been modified outside approved processes, and manual\nchanges leave incomplete records. When every change is a new artifact, version\ncontrol and the CI/CD logs show what changed, when, and who approved it. A\nsecurity patch follows the same path as a feature release: it becomes a new\nartifact that replaces the running one, so it is built, tested and recorded the\nsame way.\n\nNone of this is free. The framework notes that starting a virtual machine takes\nlonger and consumes more resources than starting a container, while containers\nneed an orchestrator to run them. Every change, even a one-line setting, now\nmeans a full build and a replacement, so the build has to be automatic and quick\nenough that nobody is tempted to skip it. State has to be moved off the\nreplaceable machines before the first replacement, not after.\n\n## What it looks like in practice\n\nA Packer template describes how to build a machine image. This one is based on\nthe example in the documentation for Packer's Amazon EBS builder, with a `build`\nblock added in the form shown on the build block reference:\n\n```hcl\nsource \"amazon-ebs\" \"web\" {\n  access_key    = var.aws_access_key\n  secret_key    = var.aws_secret_key\n  region        = \"us-east-1\"\n  source_ami    = \"ami-fce3c696\"\n  instance_type = \"t2.micro\"\n  ssh_username  = \"ubuntu\"\n  ami_name      = \"web {{timestamp}}\"\n}\n\nbuild {\n  sources = [\"source.amazon-ebs.web\"]\n\n  provisioner \"shell\" {\n    scripts = fileset(\".\", \"scripts/{install,secure}.sh\")\n  }\n}\n```\n\nThe `source` block names the builder and the base image, `source_ami`, whose\nroot volume is copied and provisioned on a running instance. The `build` block\nsays which sources to build and how to provision them. The shell provisioner\nruns the two scripts inside that running machine before it is turned into a\nstatic artifact, and the artifact is a new image. The declarations of the two\nvariables are left out here. One command runs every build in the template and\nproduces the artifacts:\n\n```sh\npacker build web.pkr.hcl\n```\n\nDeployment is the second half. HashiCorp's framework puts the two tools side by\nside: images are built with Packer, deployed with Terraform, and replaced\nentirely when they need an update. When a change cannot be made in place,\nTerraform's default, according to its lifecycle reference, is to destroy the\nexisting object and then create the replacement. Updates in this flow do not require logging in to\na running production server. A change to the scripts produces a new image, the image\nproduces new machines, and the old ones are removed.\n\nThe same discipline scales down to one host. The portfolio entry on\n[running containers on Podman and Quadlet](/portfolio/ran-containers-on-podman-and-quadlet/)\nkeeps provisioning and deployment apart. Root prepares the deployment path a\nsingle time; after that, releases are made by an ordinary user with no special\nrights, who may run exactly one script and nothing else. A release is a fresh\nimage followed by a restart of the service, without a root shell or free-form\ncommands. The entry on\n[CI/CD with GitHub Actions](/portfolio/built-github-actions-ci-cd-pipelines/)\ngoes further on the image side: the production frontend runs on a distroless\nbase that has neither a shell nor a package manager, and a build is promoted\nfrom one environment to the next by the workflows, not by a person. An image\nlike that leaves almost nothing to log in to and no tool to install anything\nwith, so building its successor is the only sensible way to change it.\n\nThe entry on\n[servers you can rebuild from scratch](/portfolio/servers-you-can-rebuild-from-scratch/)\nis a careful example of the other model. There, Ansible brings a running host to\nthe state written in the repository, each play is idempotent, and the repository\nsupplies the configuration for rebuilding the declared environment. The entry is open about what\nin-place change costs. Any change takes longer than a direct edit on the server,\nand a run that stops halfway leaves the host worse off than a run that declines\nto start; that is the reason a preflight play now runs ahead of it. A host left\nhalf-changed is a step towards the fragile server Fowler described, and\nreplacing the machine instead of changing it is the other way around that risk.\n\n## Where it goes wrong\n\nThe model is simple to state and easy to erode. Most failures come from one\nexception that seemed harmless at the time.\n\n**A hot fix over SSH brings drift back.** Under pressure, someone logs in to a\nrunning server and edits a file. The fix works until the next deployment\nreplaces the machine, and then it is gone, along with any record of why it was\nneeded. Until then, the running server no longer matches its image, which is\nexactly the condition the approach exists to prevent. The fix is to send every\nchange, the urgent one included, through the image build, and to keep that build\nfast enough to be the quickest route. HashiCorp's guidance rules out connecting\nto a live server for hand edits or updates altogether.\n\n**Data written to the machine is deleted with it.** Docker's documentation says\nthat when a container is deleted, the file changes made inside it are deleted\ntoo. Uploads, generated files or a local database on a replaceable machine\ndisappear at the next release. The fix is to keep state outside the part that\ngets replaced. Docker volumes persist data beyond the lifecycle of a single\ncontainer, and a new container attached to the same volume finds the files still\nthere; for machines, the same role falls to a separate disk or a database\nservice that is not part of the image.\n\n**Replacement takes the service down.** Terraform's default order is to destroy\nthe existing object first and create the replacement second, which leaves a\nwindow with nothing running. The `create_before_destroy` setting in a resource's\n`lifecycle` block reverses that order:\n\n```hcl\nresource \"aws_instance\" \"web\" {\n  # ...\n  lifecycle {\n    create_before_destroy = true\n  }\n}\n```\n\nHashiCorp makes this opt-in because two copies of one object cannot always exist\nside by side: many object types demand a unique name or carry other limits. Its\nreference asks you to learn each resource type's constraints before relying on\nthe setting.\n\n**A rebuild quietly starts from a newer base.** Instead of a fixed `source_ami`,\nthe Amazon EBS builder can fill it from a `source_ami_filter`, and with\n`most_recent` set to true the filter selects the newest matching image, which\nthe documentation calls most useful for a daily distro build. That is convenient\nfor staying current, but two builds of the same template on different days then\nstart from different bases, and rebuilding an old release no longer reproduces\nit. The fix for release images is to pin `source_ami` to a specific image and\nmove it forward deliberately, in a reviewed commit.\n\n**Secrets are baked into the image.** HashiCorp's framework is firm that\nartifacts must never contain secrets such as database passwords, API keys or\nencryption keys: whoever can read the artifact can pull them out, and changing a\ncredential then forces a rebuild and redeploy of every artifact. The fix is to\nfetch secrets when the machine or container starts. HashiCorp's own pattern uses\nVault, where a starting machine authenticates and receives temporary credentials\nthat expire, and credentials are rotated in Vault without rebuilding anything.\nThe template above follows the same rule for its own keys, which arrive as\nvariables rather than as text in the file.\n\n**The previous artifact is gone when it is needed.** Rollback means redeploying\nthe previous artifact, which assumes it still exists. A cleanup job that deletes\nold images, or a habit of reusing one image name for every build, turns a\nrollback into a rebuild from old source on a base that may have moved. The fix\nis to keep the images of recent releases that ran in production, give each one a\nname that is never reused, and rehearse a rollback to one of them before the day\nit matters.\n\n## Further reading\n\n- [Create immutable infrastructure](https://developer.hashicorp.com/well-architected-framework/define-and-automate-processes/define/immutable-infrastructure)\n  — the Well-Architected Framework on drift, rollback, image types and secrets.\n- [Trash Your Servers and Burn Your Code: Immutable Infrastructure and Disposable Components](https://web.archive.org/web/20160418161330/http://chadfowler.com/2013/06/23/immutable-deployments.html)\n  — Chad Fowler's June 2013 post, read through an Internet Archive copy.\n- [Introduction to Packer](https://developer.hashicorp.com/packer/docs/intro)\n  — what Packer builds and what a machine image contains.\n- [Amazon EBS builder](https://developer.hashicorp.com/packer/integrations/hashicorp/amazon/latest/components/builder/ebs)\n  — the builder used above, with `source_ami` and `source_ami_filter`.\n- [lifecycle meta-argument reference](https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle)\n  — Terraform's destroy-then-create default and `create_before_destroy`.\n- [Persisting container data](https://docs.docker.com/get-started/docker-concepts/running-containers/persisting-container-data/)\n  — why files written inside a container go with it, and how volumes keep them.\n\n\u003chttps://engineer.company/glossary/immutable-infrastructure/\u003e\n","date_modified":"2026-11-12T05:34:23Z","description":"What immutable infrastructure means, why teams replace servers instead of patching them, how an image is built and swapped in, and where the model breaks.","formats":{"gemtext":"https://engineer.company/glossary/immutable-infrastructure/index.gmi","gopher":"https://engineer.company/glossary/immutable-infrastructure/gophermap","html":"https://engineer.company/glossary/immutable-infrastructure/","json":"https://engineer.company/glossary/immutable-infrastructure/index.json","markdown":"https://engineer.company/glossary/immutable-infrastructure/index.md","pdf":"https://engineer.company/glossary/immutable-infrastructure/index.pdf"},"kind":"page","language":"en","title":"What is immutable infrastructure?","url":"https://engineer.company/glossary/immutable-infrastructure/","version":1}