Built dead‑man's‑switch monitoring that pings only while memory and disk are healthy, so a degraded host raises an alert by going silent — and caught six variable names saying "free" where the check correctly measured "available", an order of magnitude apart on a 464 MB box.
Work carried out: 2025
Situation. A 464 MB host running a forge, a database and a web server has two realistic ways to die: it runs out of memory, or it runs out of disk. Neither announces itself. Both are entirely predictable a few hours in advance if anything is looking, and nothing was.
Task. The host needed an alarm that works when the host does not — which rules out anything that has to send a message at the moment of failure.
Action. The answer is a dead‑man’s switch on a timer. Every fifteen minutes with jitter, a small script measures available memory and free disk and pings an external service only if both are above their thresholds. Silence is the alert. A machine that has run out of memory, lost its network or stopped booting produces exactly the same signal as one that is unhealthy, which is the correct behaviour and the reason this shape was chosen over an agent that reports a status. The measurement is available memory, not free memory, and that distinction turned into the most instructive part of the work: the script had been reading the right column all along while six variable names around it said “free”. On a healthy Linux box free memory is near zero because the kernel uses idle memory for cache, so a reader checking those names against a ten per cent threshold would see twelve megabytes free on a 464 MB machine, conclude the check was broken, and fix it with a one‑character change that converts a working alarm into one that breaches permanently and gets muted inside a week.
Result. The host now has an alarm whose failure mode is to fire, and the naming that would have destroyed it has been corrected. The switch’s real limitation is stated in its own documentation: it proves the box is healthy, not that the site is serving, and those are different questions.