# Container runtime ## What it is A container runtime is the software between a container image on disk and an isolated process in the kernel. The phrase covers two layers, and people often mean different ones: the low-level runtime, also called the OCI runtime, and the high-level runtime, more often called a container engine. The low-level runtime does one narrow job. It reads a bundle, which the Open Container Initiative's runtime specification defines as a directory holding a `config.json` file and the container's root filesystem, asks the kernel for the namespaces, cgroups and mounts that the configuration describes, and starts the process inside them. The specification names the operations every runtime must support, namely state, create, start, kill and delete, and leaves the command line to each implementation. Version 1.0 was published in July 2017. runc, maintained by the OCI itself, calls itself "a low level tool not designed with an end user in mind". crun does the same job in C with a smaller memory footprint and is the default Podman's documentation names. youki is a third, written in Rust. The engine sits above it. containerd describes itself as a daemon that manages the complete container lifecycle of its host: image transfer and storage, container execution and supervision, and the storage and network attachments around them. CRI-O does a similar job built for Kubernetes alone. An engine pulls and unpacks the image, writes the bundle, hands it to the low-level runtime, and keeps watching the process after that runtime has exited. Docker Engine sits one step higher still: its documentation says it uses containerd for the container lifecycle, and that containerd uses runc by default. Podman takes the other route and keeps no long-running daemon. Each command prepares the bundle itself, calls crun or runc, and leaves a small monitor called conmon in charge of the running container. Kubernetes adds one more piece of vocabulary. The kubelet on each node talks to the engine over the Container Runtime Interface, CRI, a gRPC protocol that containerd and CRI-O both implement. Docker Engine predates CRI, so for years Kubernetes carried an adapter for it called dockershim, and removed it in v1.24. ## Why it matters Choosing a container runtime is really choosing which layer to change, and the two layers buy different things. The engine decides how containers live from day to day: who pulls and stores images, who restarts a crashed container, where its logs go, which API other tools speak, and who owns the process tree. Swapping Docker Engine for Podman on a server, or for containerd on a Kubernetes node, changes all of that and leaves the containers themselves alone, because the image format and the bundle are the same OCI standards underneath. The Kubernetes documentation on the dockershim removal says it plainly: images produced by `docker build` work with all CRI implementations. The low-level runtime decides how isolated the process is. runc and crun start an ordinary process on the host's kernel, which is fast and fully compatible, and means every container on the machine shares that one kernel. gVisor's runtime, runsc, puts an application kernel written in Go between the container and the host and answers the container's system calls itself; its own documentation names the price as reduced application compatibility and higher overhead per system call. Kata Containers boots each workload inside a lightweight virtual machine with a dedicated kernel, using hardware virtualization as a second layer of defence, and needs a hypervisor on each node that runs it. Kubernetes lets the second choice be made per pod rather than per cluster. A RuntimeClass names a handler configured in the engine, and a pod that sets `runtimeClassName` runs under that handler, so an untrusted workload can sit in gVisor or Kata while everything else stays on runc. The documentation frames this as a balance of performance against security. ## What it looks like in practice In everyday speech the phrase drifts towards the engine layer, and the work on this site uses it in that sense. Twenty Docker-packaged applications [spread over two clouds and a few container runtimes](/portfolio/deployed-and-maintained-20-docker-containerized-applications-troubleshooting-19/) had to deploy consistently and be diagnosable fast when something broke. Docker packaged them, troubleshooting went through Podman, and orchestration debugging went through Kubernetes. Working across engines like that leans on the layering above: the engines differ, and the OCI image and runtime standards beneath them are shared. On a newer host the choice was made at the engine layer. [The container plane runs on Podman and Quadlet in place of Docker](/portfolio/deployed-a-container-plane-on-podman-and-quadlet-122/), because the default engine publishes container ports above the firewall rules the hardening role installs, and because it would add a second supervisor beside systemd. What the choice changed is who starts each container and who watches it afterwards: containers became services that start the way services start, under the supervisor the host already trusted. The same reasoning carries over to [a 512 MB host that runs the whole company](/portfolio/ran-the-whole-company-on-one-512mb-host-109/), where memory settled every argument. Its container plane is Quadlet units under systemd rather than a daemon with its own supervisor, a choice made on a machine where about 464 MB of memory is usable. ## Where it goes wrong Runtime trouble tends to surface at the seams between layers, where each side assumes the other is set up the way it expects. **Node tooling goes blind after a cluster upgrade.** From Kubernetes v1.24 the kubelet talks to containerd or CRI-O directly, so containers the cluster schedules never pass through Docker Engine. `docker ps` on the node lists only what Docker itself started, and monitoring agents, startup scripts and privileged pods that call the Docker CLI or mount `docker.sock` lose their view. The workloads keep running. It is the tools around them that need moving to `crictl` or the engine's own API. **The kubelet and the runtime manage cgroups two different ways.** Both must use the same cgroup driver, `cgroupfs` or `systemd`, and the kubelet defaults to `cgroupfs`. On a host where systemd is the init system, that leaves two cgroup managers with two views of the same resources, and the Kubernetes documentation warns that such nodes can become unstable under resource pressure. The fix is the systemd driver on both sides, which is also the recommended driver with cgroup v2. For containerd 2.x with runc, the runtime side looks like this: ```toml # /etc/containerd/config.toml, containerd 2.x [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options] SystemdCgroup = true ``` The kubelet side is `cgroupDriver: systemd` in its configuration, and containerd needs a restart afterwards. containerd 1.x spells the section differently, and the Kubernetes page gives both forms. Changing the driver on a node that has already joined a cluster is delicate: pods created under one driver can hit errors when they are recreated under the other, so the documented advice is to replace or reinstall the node with the new configuration. **A pod moved to a sandboxed runtime stops at startup.** A RuntimeClass whose handler the node's engine lacks sends the pod straight to the Failed phase, with the reason in an event. Past that, the workload has to suit the sandbox. gVisor answers system calls in its own kernel, so software that leans on less common kernel features may behave differently, and its documentation names system-call-heavy workloads as a poor fit. Kata needs hardware virtualization on the node, which on a cloud virtual machine means nested virtualization. A `scheduling` node selector on the RuntimeClass keeps such pods on the nodes that can run them. **An image imported by hand stays invisible to the kubelet.** containerd keeps image names and metadata per namespace. Its `ctr` client works in the namespace `default` unless told otherwise, while the CRI plugin the kubelet talks to works in `k8s.io`. An image loaded with a plain `ctr images import` therefore lands where the kubelet never looks, and the pod goes back to the registry or stops with an image pull error. Importing with `ctr -n k8s.io images import`, and checking with `crictl images`, which sees what the kubelet sees, avoids the gap. ## Further reading - [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec) — the standard every low-level runtime implements: the bundle, `config.json` and the lifecycle operations. - [runc](https://github.com/opencontainers/runc) — the OCI's own runtime, with notes on how higher-level tools drive it. - [Container Runtimes](https://kubernetes.io/docs/setup/production-environment/container-runtimes/) — cgroup drivers, CRI versions and the containerd and CRI-O settings Kubernetes expects. - [Container Runtime Interface (CRI)](https://kubernetes.io/docs/concepts/containers/cri/) — the gRPC protocol between the kubelet and the engine. - [Runtime Class](https://kubernetes.io/docs/concepts/containers/runtime-class/) — picking a different runtime handler per pod, and keeping those pods on nodes that support it. - [Alternative container runtimes](https://docs.docker.com/engine/daemon/alternative-runtimes/) — how Docker Engine hands containers to gVisor, Kata Containers, youki or Wasmtime.