Health check

What it is

A health check is a small test, repeated on a schedule, that asks a
running service whether it can do its job, together with a rule for
what happens when the answer is no. The test might be an HTTP
request to a dedicated path, a command run inside a container, or an
attempt to open a TCP port. The rule is what turns it from a
measurement into a control: a failed check can take a server out of
rotation, restart a container, stop a deployment or wake someone up.

Docker builds the idea into the image. The `HEALTHCHECK` instruction
in a Dockerfile names a command that Docker runs inside the
container at a fixed interval. Exit status 0 means healthy, 1 means
unhealthy, and 2 is reserved and must not be used. A container with
a check begins in the state `starting`, becomes `healthy` as soon as
one check passes, and turns `unhealthy` after a set number of
consecutive failures. The defaults are a check every 30 seconds, a

30-second timeout and three retries; a run that exceeds its timeout
counts as a failure, and its process is killed. An optional start
period gives a slow application time to boot, and failures inside it
are not counted. A Dockerfile holds one such instruction, the last
one wins if there are several, and `HEALTHCHECK NONE` switches off a
check inherited from the base image.

Kubernetes splits the question into three probes, each with its own
consequence. A startup probe asks whether the application has
finished starting; until it succeeds, the other two are held back,
and if it keeps failing the container is killed and handled by its
restart policy. A liveness probe asks whether the container should
be restarted, and is meant for states such as a deadlock, where the
process runs but makes no progress. A readiness probe asks whether
the container should receive traffic. When it fails, nothing is
restarted: the Pod's address is taken out of the endpoints of every
Service that matches it, and requests go to the other replicas until
the probe passes again.

Each probe uses one of four mechanisms. An `httpGet` probe passes on
any status code from 200 up to, but not including, 400. A
`tcpSocket` probe passes if the port accepts a connection, even one
that is closed straight away. A `grpc` probe passes when the
service's gRPC health check answers `SERVING`. An `exec` probe runs
a command in the container and passes on exit status 0. Unless told
otherwise, Kubernetes probes every 10 seconds, waits one second for
an answer, and acts after three failures in a row.

Load balancers add a second kind. The checks above are active:
something sends a test request whether or not users are around. A
passive check watches real traffic instead. Open-source nginx works
this way: when a request to an upstream server ends in an error,
nginx marks that server as failed and avoids it for a while, and
with the default `max_fails` of 1 a single failed attempt is enough.
The nginx documentation lists application health checks among the
features of the paid NGINX Plus subscriptions. A passive check costs
nothing extra, but it learns about a broken server from a user's
failed request, and it says nothing about a server that receives no
traffic.

Why it matters

A running process is not a working service. Docker's reference gives
the typical case: a web server caught in an endless loop, no longer
taking new connections, although its process is still running.
Anything that only watches whether the process exists, such as a
systemd unit with a restart policy, sees nothing wrong. Only a
question asked through the same door the users use can tell the
difference, and that is the whole case for a health check: it
replaces "is it running?" with "does it answer?".

Two decisions make a check useful or dangerous. The first is what
the check tests, which amounts to a definition of "working" written
in code. A handler that returns 200 without touching anything proves
the web framework is up; a handler that also queries the database
proves the database is reachable as well, and fails whenever it is
not. The second is what happens on failure, which sets the blast
radius. Taking one replica out of rotation is cheap and reversible.
Restarting it throws away its in-flight work and its warm caches,
and if every replica fails the same check at the same moment,
restarting all of them turns a slow dependency into a full outage.

Kubernetes warns about exactly this. Its documentation says liveness
probes must be configured so that they truly indicate an
unrecoverable failure, and that a wrong implementation can cause
cascading failures: containers restarted under high load, failed
client requests, and more work landing on the pods that remain. The
advice that follows is to keep the two questions apart. When an
application depends strictly on back-end services, its liveness
probe should pass as long as the application itself is healthy,
while its readiness probe also checks each required back end.
Traffic then avoids a Pod that could only answer with errors, and
the Pod is not killed for a fault that lies somewhere else.

Health checks also serve monitoring. An endpoint that a load
balancer polls can be polled from outside as well, and the alert
that fires when it fails is often the first sign of an incident.
When Irakli Nadareishvili wrote the Internet-Draft Health Check
Response Format for HTTP APIs, there was no standard for what such
an endpoint returns, and most applications relied on the HTTP status
code alone or on a format of their own. The draft proposed a JSON
body with the media type `application/health+json` and a mandatory
`status` field of `pass`, `fail` or `warn`, where `pass` and `warn`
go with a 2xx or 3xx status code and `fail` with a 4xx or 5xx. Its
last revision, from October 2021, expired in April 2022, and the
IETF lists it as not endorsed and without formal standing. It is
useful as a shared vocabulary, not as a standard.

What it looks like in practice

In a Dockerfile, the check ships with the image it checks. Here the
application's own small probe program requests a path on localhost
and exits with 0 or 1:

```dockerfile
HEALTHCHECK --interval=15s --timeout=3s --start-period=60s --retries=3 \
  CMD ["/app/bin/probe", "http://127.0.0.1:8080/livez"]
```

Docker runs the probe every 15 seconds and gives each run three
seconds. Failures during the first minute do not count unless a
check has already passed, and on Docker Engine 25.0 or later the
checks in that minute run every five seconds, the default start
interval, so a quick start is noticed quickly. After three failures
in a row the container is marked `unhealthy`. The first 4096 bytes
of the probe's output are kept for `docker inspect`, and every
change of status emits a `health_status` event. The reference
describes a status, not an action: something else has to act on it.

In Kubernetes, the three probes can be given different paths, so
that each asks its own question:

```yaml
containers:
  - name: api
    ports:
      - containerPort: 8080
    startupProbe:
      httpGet: { path: /livez, port: 8080 }
      periodSeconds: 5
      failureThreshold: 24 # up to 2 minutes to start
    livenessProbe:
      httpGet: { path: /livez, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 6 # about a minute before a restart
    readinessProbe:
      httpGet: { path: /readyz, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2
```

`/livez` answers for the process alone, and `/readyz` also checks
the database and anything else a request cannot do without. The
startup probe uses the liveness path, as the Kubernetes
documentation recommends, and allows 24 × 5 seconds before the
container is given up on. Once it passes, liveness takes over with a
long fuse, and readiness with a short one: about ten seconds of
failures take the Pod out of the Service, and a minute of them
restarts it. The defaults would allow three failures ten seconds
apart, which is too little for an application that needs two minutes
to load its data. The documentation's own rule is that a container
which usually takes longer than the initial delay plus
failureThreshold × periodSeconds to start needs a startup probe.

A health check can also be turned inside out. In dead-man's-switch
host monitoring, from September 2026, a 464 MB host running a forge
and a web server checks itself every fifteen minutes, with jitter: a
small script measures available memory and free disk and pings an
external service only when both are above their thresholds. Silence
is the alert. A host that has run out of memory, lost its network or
stopped booting sends the same signal as one that is merely
unhealthy, which is why this shape was chosen over an agent that
reports a status. The check reads available memory rather than free
memory, because the kernel fills idle memory with cache, and six
variable names that said "free" were corrected so that nobody would
"fix" a working alarm into one that breaches permanently. A received
signal confirms that the script ran and its available-memory and
free-disk thresholds passed; it does not certify general host health
or that the site is serving. Monitoring can expose warning signs,
although the timing depends on the workload and the conditions
observed.

A check at startup answers yet another question. In email with
failover and delivery logging, from 2025, mail runs through three
providers with automatic failover, and in front of it a startup
check sends a real message through the whole stack, behind a flag.
In development a failure logs a warning and the application carries
on; in staging and production it is fatal, and the process exits
rather than deploy a build that cannot send mail. The page sums up
the result as a whole category of silent failure moved from "a user
notices days later" to "the deploy stops".

Where it goes wrong

The check proves only that the process answers. A handler that
returns 200 without touching anything stays green while the database
is unreachable, the disk is full or every real request ends in an
error. Let the readiness check exercise what a request needs,
cheaply, and add a check from outside that follows a user's path
through DNS, TLS and the load balancer, since the kubelet sends its
probes straight to the Pod's own IP address and sees none of that.

Liveness depends on something outside the container. If the liveness
probe queries the database, a slow database fails it in every
replica at once, and Kubernetes restarts them all at the moment the
service is already struggling. Let liveness answer for the process
alone, as the Kubernetes documentation advises, and put dependencies
in readiness, where a failure diverts traffic instead of killing
containers.

The timings do not fit the application. Kubernetes waits one second
for an answer by default, so an endpoint that slows down under load
fails while the application is merely busy, and a container that
starts more slowly than its liveness probe allows is killed before
it is ready, again and again. Measure how long the endpoint takes
under load and set the timeout above it, and give slow starters a
startup probe sized to the slowest real start.

A redirect passes. An HTTP probe in Kubernetes counts any status
from 200 to

399 as success, so a path that redirects to a login page looks
healthy. If the redirect points to another host, the kubelet does
not follow it and records the probe as successful, with a warning.
With the HTTPS scheme the kubelet also skips certificate
verification, so an expired certificate never fails a probe. Point
probes at a path that answers 200 directly and returns a 5xx when
something is wrong, and check certificates from outside.

The health endpoint says too much to too many. A detailed response
that lists every dependency, version and host name informs an
attacker as well as a load balancer, and the Internet-Draft's
security considerations note that health endpoints may need
authentication and role-based access control. Keep the public answer
to a status, and serve the details on an internal port or to
authenticated callers only.

The check costs more than it should. Kubernetes warns that `exec`
probes fork processes on every run, which on densely packed nodes
with short intervals can add CPU overhead to the node, and a probe
that runs a heavy query every few seconds on every replica is load
in its own right. Prefer HTTP, TCP or gRPC probes, keep the endpoint
cheap, and cache the result of an expensive dependency check for a
few seconds instead of repeating it for every probe.

Further reading

  * Dockerfile reference: HEALTHCHECK — the instruction, its options
and defaults, and what each exit status means.

  * Liveness, Readiness, and Startup Probes — what each probe type
does, the four check mechanisms and every configuration field with
its default.

  * Configure Liveness, Readiness and Startup Probes — worked
examples, including a startup probe that gives a slow application
five minutes.

  * Health Check Response Format for HTTP APIs — the expired
Internet-Draft behind `application/health+json` and its pass, fail
and warn statuses.

  * Using nginx as HTTP load balancer — the passive, in-band health
checks in open-source nginx and the `max_fails` and `fail_timeout`
settings.

hThis page on the web	URL:https://engineer.company/glossary/health-check/	engineer.company	70
