Skip to content
Engineer
Menu
Design
English

Engineer / Portfolio

Built dead‑man's‑switch host monitoring

Built dead‑man's‑switch monitoring that pings only while memory and disk are healthy, so a degraded host raises an alert by going silent — and caught six variable names saying "free" where the check correctly measured "available", an order of magnitude apart on a 464 MB box.

Situation. Two important failure risks for a 464 MB host running a forge and a web server are memory exhaustion and a full disk. Monitoring can expose warning signs, although the timing depends on the workload and the conditions observed.

Task. The host needed an alarm that works when the host does not — which rules out anything that has to send a message at the moment of failure.

Action. The answer is a dead‑man’s switch on a timer. Every fifteen minutes with jitter, a small script measures available memory and free disk and pings an external service only if both are above their thresholds. Silence is the alert. A machine that has run out of memory, lost its network or stopped booting produces exactly the same signal as one that is unhealthy, which is the correct behaviour and the reason this shape was chosen over an agent that reports a status. The measurement is available memory, not free memory, and that distinction turned into the most instructive part of the work: the script had been reading the right column all along while six variable names around it said “free”. On a healthy Linux box free memory is near zero because the kernel uses idle memory for cache, so a reader checking those names against a ten per cent threshold would see twelve megabytes free on a 464 MB machine, conclude the check was broken, and fix it with a one‑character change that converts a working alarm into one that breaches permanently and gets muted inside a week.

Result. The host gained an alarm that fires when the expected signal is absent, and the misleading names were corrected. A received signal confirms that the script ran and its available‑memory and free‑disk thresholds passed; it does not certify general host health or that the site is serving.

Contact

Still working out what you need?

Tell us what is going wrong, or what you are trying to build. We will tell you what it would take, roughly what it would cost, and whether we are the right people for it.

From 1,400 DKK an hour, excluding VAT — and less per hour on a longer booking.

welcome@engineer.company +45 50 10 61 84 Copenhagen, Denmark