{"canonical_url":"https://engineer.company/glossary/kernel/","content_markdown":"# Kernel\n\n## What it is\n\nA kernel is the program at the centre of an operating system that runs with the\nprocessor's full privileges and controls access to the hardware. It decides\nwhich program runs next, hands out memory and keeps each program's memory apart\nfrom the rest, reads and writes disks through file systems, moves packets\nthrough the network stack, and enforces who may touch which file, port or\ndevice. Everything else runs in user space with fewer privileges and asks the\nkernel to act for it.\n\nThose requests are system calls, which the Linux manual calls “the fundamental\ninterface between an application and the Linux kernel”. Programs rarely make\nthem by hand: they call wrappers in a C library such as glibc, which pass the\narguments on and turn the kernel's answer into a return value and an error code.\nWhen a program stops with “Permission denied”, that wording is usually the\nkernel's answer to one of these calls, translated by the library.\n\nKernels differ in how much they keep inside. Linux keeps device drivers, file\nsystems and the network stack in the kernel itself and can load more as modules\nwhile it runs, a design usually called monolithic. Every driver then runs with\nthe kernel's full privileges. A microkernel keeps only the minimum. seL4\nprovides mechanisms for controlling access to the physical address space,\ninterrupts and processor time, and runs drivers, file systems and network stacks\nas ordinary user-mode programs; its verified 64-bit RISC-V version is about\n10,000 lines of code, with a formal proof that the C implementation does what\nits specification says. The macOS kernel, XNU, is a hybrid that combines the\nMach microkernel with parts of FreeBSD.\n\nA kernel update normally takes effect only when the machine boots into it, and\nthe kernel's version number, such as 6.12, is separate from the release number\nof the distribution around it.\n\n## Why it matters\n\nFor a business, the kernel settles three questions: how far apart its workloads\nreally are, what a security fix costs in downtime, and where the truth about a\nmachine is kept.\n\nIsolation first. A container is a set of ordinary processes that the kernel has\nplaced in namespaces, which give each group its own view of process IDs, network\ninterfaces, mounts and users, and in control groups, which share out processor\ntime, memory and disk input and output. Every container on a host runs on the\nhost's one kernel, so a flaw in that kernel is a flaw under all of them at once,\nand an image that brings its own libraries still relies on whatever kernel\nfeatures the host has. A virtual machine boots a kernel of its own. For one\nteam's services, containers are a reasonable boundary; for customers who do not\ntrust each other, give each tenant its own kernel.\n\nFixes come next. A patched kernel package on disk changes nothing until the\nmachine boots into it, and on a server that restart is the cost. Livepatching\nexists because, in the kernel documentation's words, users are “reluctant to\nreboot”: it redirects individual functions to fixed versions while the system\nruns. Only functions the kernel can trace can be patched that way, so\nlivepatching buys time for an urgent fix rather than replacing reboots. A plan\nwith a regular, rehearsed reboot costs less than one that assumes the reboot\nnever has to happen.\n\nLast, the kernel holds the live state. A firewall file describes rules; the\nrules in the kernel's packet filter decide a packet's fate. A file in\n`/etc/sysctl.d` describes a setting; the value under `/proc/sys` is the one in\nforce. A service can report itself healthy whether or not its action ever\nreached the kernel. Checks that read the kernel's own state catch failures that\na review of configuration files would pass.\n\n## What it looks like in practice\n\nA hardening review of a production host in August 2026 asked whether its SSH\nbrute-force protections worked. Each was configured and appeared in every\nreport, so the review\n[checked them against what the kernel actually does with a packet](/portfolio/three-locks-on-the-door-none-of-them-locked/)\nrather than against the files describing what should happen. Seen from the\nkernel, none of the three had acted once since the host was built. The jail's\nbans named port 22, where no socket listened after SSH had moved to port 1986.\nThe rate limit did reach the packet filter, which evaluates rules top down and\nstops at the first match, so a broader rule above it took every packet before\nthe limit was consulted. The ban action shelled out to a packet-filter program\nthat its package lists as recommended rather than required; on a host without\nit, the jail started, counted and decided, then failed at the one moment it had\nto hand a rule to the kernel. Two rules came out of the review: a security\ncontrol is proved by an assertion, which a comment cannot do, and a firewall is\nchecked by the order in which the kernel holds its rules, rather than by whether\na rule exists.\n\nThe ban path then got a test that runs every time the configuration is applied.\nEach run bans an address from a documentation-reserved block,\n[reads the resulting rule back from the kernel's packet filter](/portfolio/proved-the-intrusion-ban-works-every-run/)\nand then lifts the ban in a cleanup step that runs either way. The read-back is\nwhat makes it a test: the banning service's own status would report success for\na ban that produced no rule. A reserved address belongs to no real network, so a\nban left behind by a failed run locks nobody out. The cost: a few seconds per\nrun, and a firewall rule written to production and withdrawn each time.\n\nA 464 MB host with no room for a metrics stack got\n[seven read-only roles that render the live machine to Markdown](/portfolio/built-7-read-only-host-reporting-roles/);\nits facts snapshot computes processor utilisation from two samples of the\nkernel's own counter. The security report prints the firewall rules in the order\nthey are matched, beside the live ban rules, and each of the seven roles prints\nonly figures its own run read, the kernel's counters among them, never an\nestimate to fill a gap. Those reports turned up two security controls that were\ndead and a listening port that nobody had declared.\n\n## Where it goes wrong\n\n**A kernel setting changed by hand reverts after the next reboot.** `sysctl -w`\nwrites the value into the running kernel only, so it lasts until restart. The\nopposite slip: a file under `/etc/sysctl.d` is read at boot, so the change\nwaits, unannounced, for some later restart. Write the setting to a file, load it\nnow, and read the live value back:\n\n```sh\necho 'vm.swappiness = 10' | sudo tee /etc/sysctl.d/90-local.conf\nsudo sysctl --system\nsysctl vm.swappiness\n```\n\n**A CPU graph stays flat through a load spike.** The `cpu` line in `/proc/stat`\ncounts time since boot, in units of USER_HZ (hundredths of a second on most\narchitectures), so a single reading gives the average since boot, which barely\nmoves on a machine up for months. Read it twice and divide the differences:\n\n```sh\nhead -1 /proc/stat; sleep 5; head -1 /proc/stat\n```\n\nBusy time is the change in user, nice, system, irq and softirq, idle time is the\nchange in idle and iowait, and utilisation is busy over busy plus idle. The\nmanual calls iowait unreliable, so treat it as a hint rather than an alert.\n\n**A crash report comes back with a request to reproduce it on an untainted\nkernel.** The kernel marks itself tainted when something happens that may matter\nto a later investigation, such as loading a proprietary, out-of-tree or unsigned\nmodule, and its documentation warns that developers often ignore bug reports\nfrom tainted kernels. The mark stays until reboot, even after the module is\nunloaded. Read it before filing:\n\n```sh\ncat /proc/sys/kernel/tainted\n```\n\nZero means untainted; any other number is a sum of flags, among them `1` for a\nproprietary module, `4096` for an out-of-tree module and `8192` for an unsigned\none. Reproduce the fault after booting without the module that set the flag.\n\n**A new network card shows up in the hardware list and the system offers no\ninterface for it.** Drivers are part of the kernel, and a kernel older than the\nhardware may lack one. `lspci -k` shows each device's driver in use and the\nmodules able to handle it; a device without a driver-in-use line has none bound.\nInstall a newer kernel series from the distribution rather than building the\nvendor's module by hand: an out-of-tree module must be rebuilt for every new\nkernel, and loading it taints the kernel.\n\n## Further reading\n\n- [The Linux kernel user's and administrator's guide](https://docs.kernel.org/admin-guide/index.html)\n  — the kernel project's own manual for running Linux.\n- [syscalls(2)](https://man7.org/linux/man-pages/man2/syscalls.2.html)\n  — every Linux system call, with the kernel version that introduced it.\n- [namespaces(7)](https://man7.org/linux/man-pages/man7/namespaces.7.html)\n  — the kinds of namespace and how containers are built from them.\n- [proc_stat(5)](https://man7.org/linux/man-pages/man5/proc_stat.5.html)\n  — each field of `/proc/stat`, including the iowait warning.\n- [Tainted kernels](https://docs.kernel.org/admin-guide/tainted-kernels.html)\n  — each taint flag and how to decode the number.\n- [seL4 FAQ](https://sel4.systems/About/FAQ.html)\n  — what a microkernel leaves out, and the assumptions behind seL4's proof.\n\n\u003chttps://engineer.company/glossary/kernel/\u003e\n","date_modified":"2026-11-08T11:12:58Z","description":"What a kernel settles for the software above it: system calls, the isolation containers share, the reboot a fix needs, and live state over config.","formats":{"gemtext":"https://engineer.company/glossary/kernel/index.gmi","gopher":"https://engineer.company/glossary/kernel/gophermap","html":"https://engineer.company/glossary/kernel/","json":"https://engineer.company/glossary/kernel/index.json","markdown":"https://engineer.company/glossary/kernel/index.md","pdf":"https://engineer.company/glossary/kernel/index.pdf"},"kind":"page","language":"en","title":"What is a kernel?","url":"https://engineer.company/glossary/kernel/","version":1}