{"canonical_url":"https://engineer.company/glossary/health-check/","content_markdown":"# Health check\n\n## What it is\n\nA health check is a small test, repeated on a schedule, that asks a running\nservice whether it can do its job, together with a rule for what happens when\nthe answer is no. The test might be an HTTP request to a dedicated path, a\ncommand run inside a container, or an attempt to open a TCP port. The rule is\nwhat turns it from a measurement into a control: a failed check can take a\nserver out of rotation, restart a container, stop a deployment or wake someone\nup.\n\nDocker builds the idea into the image. The `HEALTHCHECK` instruction in a\nDockerfile names a command that Docker runs inside the container at a fixed\ninterval. Exit status 0 means healthy, 1 means unhealthy, and 2 is reserved and\nmust not be used. A container with a check begins in the state `starting`,\nbecomes `healthy` as soon as one check passes, and turns `unhealthy` after a set\nnumber of consecutive failures. The defaults are a check every 30 seconds, a\n30-second timeout and three retries; a run that exceeds its timeout counts as a\nfailure, and its process is killed. An optional start period gives a slow\napplication time to boot, and failures inside it are not counted. A Dockerfile\nholds one such instruction, the last one wins if there are several, and\n`HEALTHCHECK NONE` switches off a check inherited from the base image.\n\nKubernetes splits the question into three probes, each with its own consequence.\nA startup probe asks whether the application has finished starting; until it\nsucceeds, the other two are held back, and if it keeps failing the container is\nkilled and handled by its restart policy. A liveness probe asks whether the\ncontainer should be restarted, and is meant for states such as a deadlock, where\nthe process runs but makes no progress. A readiness probe asks whether the\ncontainer should receive traffic. When it fails, nothing is restarted: the Pod's\naddress is taken out of the endpoints of every Service that matches it, and\nrequests go to the other replicas until the probe passes again.\n\nEach probe uses one of four mechanisms. An `httpGet` probe passes on any status\ncode from 200 up to, but not including, 400. A `tcpSocket` probe passes if the\nport accepts a connection, even one that is closed straight away. A `grpc` probe\npasses when the service's gRPC health check answers `SERVING`. An `exec` probe\nruns a command in the container and passes on exit status 0. Unless told\notherwise, Kubernetes probes every 10 seconds, waits one second for an answer,\nand acts after three failures in a row.\n\nLoad balancers add a second kind. The checks above are active: something sends a\ntest request whether or not users are around. A passive check watches real\ntraffic instead. Open-source nginx works this way: when a request to an upstream\nserver ends in an error, nginx marks that server as failed and avoids it for a\nwhile, and with the default `max_fails` of 1 a single failed attempt is enough.\nThe nginx documentation lists application health checks among the features of\nthe paid NGINX Plus subscriptions. A passive check costs nothing extra, but it\nlearns about a broken server from a user's failed request, and it says nothing\nabout a server that receives no traffic.\n\n## Why it matters\n\nA running process is not a working service. Docker's reference gives the typical\ncase: a web server caught in an endless loop, no longer taking new connections,\nalthough its process is still running. Anything that only watches whether\nthe process exists, such as a [systemd](/glossary/#systemd) unit with a restart\npolicy, sees nothing wrong. Only a question asked through the same door the\nusers use can tell the difference, and that is the whole case for a health\ncheck: it replaces \"is it running?\" with \"does it answer?\".\n\nTwo decisions make a check useful or dangerous. The first is what the check\ntests, which amounts to a definition of \"working\" written in code. A handler\nthat returns 200 without touching anything proves the web framework is up; a\nhandler that also queries the database proves the database is reachable as well,\nand fails whenever it is not. The second is what happens on failure, which sets\nthe blast radius. Taking one replica out of rotation is cheap and reversible.\nRestarting it throws away its in-flight work and its warm caches, and if every\nreplica fails the same check at the same moment, restarting all of them turns a\nslow dependency into a full outage.\n\nKubernetes warns about exactly this. Its documentation says liveness probes must\nbe configured so that they truly indicate an unrecoverable failure, and that a\nwrong implementation can cause cascading failures: containers restarted under\nhigh load, failed client requests, and more work landing on the pods that\nremain. The advice that follows is to keep the two questions apart. When an\napplication depends strictly on back-end services, its liveness probe should\npass as long as the application itself is healthy, while its readiness probe\nalso checks each required back end. Traffic then avoids a Pod that could only\nanswer with errors, and the Pod is not killed for a fault that lies somewhere\nelse.\n\nHealth checks also serve [monitoring](/glossary/#monitoring). An endpoint that a\nload balancer polls can be polled from outside as well, and the alert that fires\nwhen it fails is often the first sign of an [incident](/glossary/#incident).\nWhen Irakli Nadareishvili wrote the Internet-Draft Health Check Response Format\nfor HTTP APIs, there was no standard for what such an endpoint returns, and most\napplications relied on the HTTP status code alone or on a format of their own.\nThe draft proposed a JSON body with the media type `application/health+json` and\na mandatory `status` field of `pass`, `fail` or `warn`, where `pass` and `warn`\ngo with a 2xx or 3xx status code and `fail` with a 4xx or 5xx. Its last\nrevision, from October 2021, expired in April 2022, and the IETF lists it as not\nendorsed and without formal standing. It is useful as a shared vocabulary, not\nas a standard.\n\n## What it looks like in practice\n\nIn a Dockerfile, the check ships with the image it checks. Here the\napplication's own small probe program requests a path on localhost and exits\nwith 0 or 1:\n\n```dockerfile\nHEALTHCHECK --interval=15s --timeout=3s --start-period=60s --retries=3 \\\n  CMD [\"/app/bin/probe\", \"http://127.0.0.1:8080/livez\"]\n```\n\nDocker runs the probe every 15 seconds and gives each run three seconds.\nFailures during the first minute do not count unless a check has already passed,\nand on Docker Engine 25.0 or later the checks in that minute run every five\nseconds, the default start interval, so a quick start is noticed quickly. After\nthree failures in a row the container is marked `unhealthy`. The first 4096\nbytes of the probe's output are kept for `docker inspect`, and every change of\nstatus emits a `health_status` event. The reference describes a status, not an\naction: something else has to act on it.\n\nIn Kubernetes, the three probes can be given different paths, so that each asks\nits own question:\n\n```yaml\ncontainers:\n  - name: api\n    ports:\n      - containerPort: 8080\n    startupProbe:\n      httpGet: { path: /livez, port: 8080 }\n      periodSeconds: 5\n      failureThreshold: 24 # up to 2 minutes to start\n    livenessProbe:\n      httpGet: { path: /livez, port: 8080 }\n      periodSeconds: 10\n      timeoutSeconds: 2\n      failureThreshold: 6 # about a minute before a restart\n    readinessProbe:\n      httpGet: { path: /readyz, port: 8080 }\n      periodSeconds: 5\n      timeoutSeconds: 2\n      failureThreshold: 2\n```\n\n`/livez` answers for the process alone, and `/readyz` also checks the database\nand anything else a request cannot do without. The startup probe uses the\nliveness path, as the Kubernetes documentation recommends, and allows 24 × 5\nseconds before the container is given up on. Once it passes, liveness takes over\nwith a long fuse, and readiness with a short one: about ten seconds of failures\ntake the Pod out of the Service, and a minute of them restarts it. The defaults\nwould allow three failures ten seconds apart, which is too little for an\napplication that needs two minutes to load its data. The documentation's own\nrule is that a container which usually takes longer than the initial delay plus\nfailureThreshold × periodSeconds to start needs a startup probe.\n\nA health check can also be turned inside out. In [dead-man's-switch host monitoring](/portfolio/built-dead-mans-switch-host-monitoring/), from September 2026, a 464 MB host running a forge and a web server checks itself every fifteen minutes, with jitter: a small script measures available memory and free disk and pings an external service only when both are above their thresholds. Silence is the alert. A host that has run out of memory, lost its network or stopped booting sends the same signal as one that is merely unhealthy, which is why this shape was chosen over an agent that reports a status. The check reads available memory rather than free memory, because the kernel fills idle memory with cache, and six variable names that said \"free\" were corrected so that nobody would \"fix\" a working alarm into one that breaches permanently. A received signal confirms that the script ran and its available-memory and free-disk thresholds passed; it does not certify general host health or that the site is serving. Monitoring can expose warning signs, although the timing depends on the workload and the conditions observed.\n\nA check at startup answers yet another question. In\n[email with failover and delivery logging](/portfolio/built-email-with-failover-and-delivery-logging/),\nfrom 2025, mail runs through three providers with automatic failover, and in\nfront of it a startup check sends a real message through the whole stack, behind\na flag. In development a failure logs a warning and the application carries on;\nin staging and production it is fatal, and the process exits rather than deploy\na build that cannot send mail. The page sums up the result as a whole category\nof silent failure moved from \"a user notices days later\" to \"the deploy stops\".\n\n## Where it goes wrong\n\n**The check proves only that the process answers.** A handler that returns 200\nwithout touching anything stays green while the database is unreachable, the\ndisk is full or every real request ends in an error. Let the readiness check\nexercise what a request needs, cheaply, and add a check from outside that\nfollows a user's path through DNS, TLS and the load balancer, since the kubelet\nsends its probes straight to the Pod's own IP address and sees none of that.\n\n**Liveness depends on something outside the container.** If the liveness probe\nqueries the database, a slow database fails it in every replica at once, and\nKubernetes restarts them all at the moment the service is already struggling.\nLet liveness answer for the process alone, as the Kubernetes documentation\nadvises, and put dependencies in readiness, where a failure diverts traffic\ninstead of killing containers.\n\n**The timings do not fit the application.** Kubernetes waits one second for an\nanswer by default, so an endpoint that slows down under load fails while the\napplication is merely busy, and a container that starts more slowly than its\nliveness probe allows is killed before it is ready, again and again. Measure how\nlong the endpoint takes under load and set the timeout above it, and give slow\nstarters a startup probe sized to the slowest real start.\n\n**A redirect passes.** An HTTP probe in Kubernetes counts any status from 200 to\n399 as success, so a path that redirects to a login page looks healthy. If the\nredirect points to another host, the kubelet does not follow it and records the\nprobe as successful, with a warning. With the HTTPS scheme the kubelet also\nskips certificate verification, so an expired certificate never fails a probe.\nPoint probes at a path that answers 200 directly and returns a 5xx when\nsomething is wrong, and check certificates from outside.\n\n**The health endpoint says too much to too many.** A detailed response that\nlists every dependency, version and host name informs an attacker as well as a\nload balancer, and the Internet-Draft's security considerations note that health\nendpoints may need authentication and role-based access control. Keep the public\nanswer to a status, and serve the details on an internal port or to\nauthenticated callers only.\n\n**The check costs more than it should.** Kubernetes warns that `exec` probes\nfork processes on every run, which on densely packed nodes with short intervals\ncan add CPU overhead to the node, and a probe that runs a heavy query every few\nseconds on every replica is load in its own right. Prefer HTTP, TCP or gRPC\nprobes, keep the endpoint cheap, and cache the result of an expensive dependency\ncheck for a few seconds instead of repeating it for every probe.\n\n## Further reading\n\n- [Dockerfile reference: HEALTHCHECK](https://docs.docker.com/reference/dockerfile/#healthcheck)\n  — the instruction, its options and defaults, and what each exit status means.\n- [Liveness, Readiness, and Startup Probes](https://kubernetes.io/docs/concepts/workloads/pods/probes/)\n  — what each probe type does, the four check mechanisms and every configuration\n  field with its default.\n- [Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)\n  — worked examples, including a startup probe that gives a slow application\n  five minutes.\n- [Health Check Response Format for HTTP APIs](https://datatracker.ietf.org/doc/draft-inadarei-api-health-check/)\n  — the expired Internet-Draft behind `application/health+json` and its pass,\n  fail and warn statuses.\n- [Using nginx as HTTP load balancer](https://nginx.org/en/docs/http/load_balancing.html)\n  — the passive, in-band health checks in open-source nginx and the `max_fails`\n  and `fail_timeout` settings.\n\n\u003chttps://engineer.company/glossary/health-check/\u003e\n","date_modified":"2026-11-12T05:34:23Z","description":"How a running service is asked whether it can do its job, what Docker and Kubernetes do with the answer, and how a badly chosen check causes outages.","formats":{"gemtext":"https://engineer.company/glossary/health-check/index.gmi","gopher":"https://engineer.company/glossary/health-check/gophermap","html":"https://engineer.company/glossary/health-check/","json":"https://engineer.company/glossary/health-check/index.json","markdown":"https://engineer.company/glossary/health-check/index.md","pdf":"https://engineer.company/glossary/health-check/index.pdf"},"kind":"page","language":"en","title":"What is a health check?","url":"https://engineer.company/glossary/health-check/","version":1}