{"canonical_url":"https://engineer.company/glossary/monitoring/","content_markdown":"# Monitoring\n\n## What it is\n\nMonitoring is the continuous reading of what a live system reports about itself:\nsomething keeps holding those reports against lines someone drew and decides\nwhen a person needs to hear about it. Google's Site Reliability Engineering book\ndefines monitoring as the collection, processing, aggregation and display of\nquantitative data about a system in real time: how many requests arrive and of\nwhich kinds, how many fail and how, how long they take, and how long each server\nhas been running. The aim is an order of events: the system says it is in\ntrouble before a user does.\n\nThe data comes from two directions. White-box monitoring reads what a system\nexposes about its own insides, such as a metrics endpoint, a queue length or a\ncount of retries, so it can see trouble building before anyone feels it.\nBlack-box monitoring stands outside and behaves like a user: it requests a\npage, resolves a name or completes a login, and reports whether that worked and\nhow long it took. It sees only what is already broken, which is what makes it a\ntrustworthy reason to wake someone.\n\nMost of what a monitoring system keeps is numbers sampled over time, because\nnumbers are cheap to store and easy to hold against a threshold. The same SRE\nchapter names four of them as the place to start for anything that serves\nusers: latency, the time a request takes; traffic, how much demand arrives;\nerrors, the share of requests that fail; and saturation, how close the busiest\nresource is to full.\n\nAround the numbers sit two outputs. A dashboard is for looking, when someone\nalready suspects something. An alert is for interrupting, when everyone is\nbusy with something else. In the Prometheus ecosystem the server evaluates the\nrules, and Alertmanager groups, deduplicates and routes whatever fires.\n\nTwo neighbours are easy to confuse with it. Logging writes down individual\nevents, one line per thing that happened, and is what someone reads after an\nalert to find out why. Observability, as the OpenTelemetry project frames it,\nis the wider ability to ask a system new questions from the outside.\nMonitoring is the narrower and older practice: questions decided ahead of time,\nasked continuously, with an answer that can page a person.\n\n## Why it matters\n\nThe alternative is hearing about trouble from the people it hurts, at whatever\nhour they notice. Monitoring moves that moment earlier and hands it to someone\nwho can act.\n\nThe scarce resource is attention rather than software. A page interrupts work\nduring the day and sleep at night, and the SRE book is blunt about what happens\nwhen pages come too often: people skim them, second-guess them and finally\nignore them, sometimes including the real one hidden in the noise. The\nPrometheus project's guidance fits in a sentence: \"keep alerting simple, alert\non symptoms, have good consoles to allow pinpointing causes, and avoid having\npages where there is nothing to do\". Every rule that can wake someone is a\npromise that, when it fires, a person has something urgent to do.\n\nThat turns the central decision into one about what users feel. A disk at 85%\nis a cause; a checkout that fails is a symptom. Alerting on symptoms keeps the\nrule count small, because there are far fewer ways for users to be hurt than\nways for machines to misbehave. Causes still belong on dashboards and in\ntickets, where they help diagnosis without waking anyone. Capacity is the\nhonest exception: a resource close to full hurts nobody today, but it will need\na person soon.\n\nThen comes the question of running it or renting it. Running it means a\ntime-series store, an alert router and someone who keeps both healthy. A hosted\nservice takes that work away and bills by the volume sent, so every new host\nshows up on an invoice. The SRE book notes that even at Google, a team of 10–12\nengineers typically has one or two whose main job is monitoring. Either way it\nis a system with its own upkeep, to be sized to the system it watches.\n\n## What it looks like in practice\n\nTwo of the founder's earlier roles and the company's own host show the same\nidea at very different sizes.\n\nFrom 2015 to 2022, in an Infrastructure Engineer role, the job was\n[24/7 support for about 1,000 IPTV/OTT streaming servers](/portfolio/ran-24-7-support-for-1000-iptv-ott-servers/),\nplus a good number of client-owned systems, for customers in China, the US and\nGermany. With customers in three time zones there was no quiet hour, and a\nproblem at 3am local time was primetime somewhere else. Much of the work was\nnoticing something drifting before it became an outage, and problems were dealt\nwith before they reached a viewer's screen. On a service like that, success\nlooks like nothing happening, and monitoring is what lets a team know that\nrather than hope it.\n\nIn a Database Engineer role from 2022 to 2024, across\n[100 PostgreSQL, MS SQL and BigQuery databases](/portfolio/ran-100-postgresql-ms-sql-bigquery-warehouses/),\nautomated monitoring reduced downtime by 50%.\n\nMonitoring the company's own public-services host means working at the other\nend of the scale.\n[A single 512 MB machine runs those public services](/portfolio/a-whole-company-on-one-small-server/),\nwith about 464 MB of it usable, and memory is the argument that settles every\ndesign decision on it. Reporting there is a pull rather than an agent, a\npipeline and a dashboard: seven commands that read the host and render\nMarkdown, change nothing, and run only when someone asks. The onion-service\ndecision shows the method: once the question came up, the host was sampled for\na day and the readings settled it. In those recorded samples, the low point\nfor available memory was about 310 MB of 464, swap held at 2.6% and the\nprocessor idled 99.7% of the time. At that size, monitoring is a measurement\ntaken at the moment a decision needs one.\n\n## Where it goes wrong\n\n**The pager goes off so often that people stop reading it.** Each incident\nleaves a rule behind, many of them fire on causes that clear by themselves, and\nwithin months the real page gets skimmed past with the rest. Page only on what\nusers would notice, at one point in the stack, and send everything else to a\nticket queue or a dashboard. Then let the condition hold for a while before it\nfires, so a blip that clears in a minute wakes nobody:\n\n```yaml\ngroups:\n    - name: checkout\n      rules:\n          - alert: CheckoutErrorsHigh\n            expr: job:checkout_errors:ratio5m \u003e 0.02\n            for: 10m\n            labels:\n                severity: page\n```\n\n**The monitoring stops, and its silence reads as good news.** A rule that never\nfires looks exactly like a system that never breaks. If the Prometheus server,\nthe Alertmanager or the mail relay goes down, every alert stops at once.\nPrometheus's own advice is to alert on the monitoring itself, and better still\nto test the whole path end to end, so one check proves an alert can travel from\nrule to inbox. Scheduled jobs fall into the same trap: a backup that stops\nrunning raises no error, it simply stops leaving traces. Page when the last\nsuccess is too old, allowing at least two full runs; for an hour-long job that\nruns every 4 hours, the project suggests 10 hours:\n\n```promql\ntime() - backup_last_success_timestamp_seconds \u003e 10 * 3600\n```\n\n**The average says all is well while some users wait.** Averages are easy to\ngraph, and they hide the tail. The SRE book's example is a service averaging\n100 ms at 1,000 requests a second, where 1% of requests can easily take\n5 seconds; when a page depends on several such services, one backend's 99th\npercentile can become the frontend's median. Fast failures make it worse,\nbecause a quick error drags the average down. Record latency in buckets, alert\non a high percentile, and track failed requests apart from successful ones.\n\n**Every internal number is green while users cannot reach the service.**\nWhite-box metrics come from inside the system, so they share its blind spots:\nan expired certificate, a broken DNS record or a firewall change can leave\nevery one of them healthy. A black-box check from a machine outside the\nsystem's network catches what is otherwise invisible, and keeps working when\nthe internal monitoring goes down with everything else.\n\n## Further reading\n\n- [Prometheus: Overview](https://prometheus.io/docs/introduction/overview/) —\n  what a metrics-based monitoring system collects, which parts make it up, and\n  where per-event precision calls for something else.\n- [Alerting](https://prometheus.io/docs/practices/alerting/) — the Prometheus\n  project's guidance on what to page on, from online services and batch jobs to\n  monitoring the monitoring.\n- [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) —\n  how alerts are deduplicated, grouped, silenced, inhibited and routed to a\n  receiver.\n- [Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)\n  — chapter 6 of Google's Site Reliability Engineering book: symptoms and\n  causes, the four golden signals, and the tail.\n- [Observability primer](https://opentelemetry.io/docs/concepts/observability-primer/)\n  — OpenTelemetry's account of signals, SLIs and SLOs, and where observability\n  reaches past monitoring.\n\n\u003chttps://engineer.company/glossary/monitoring/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"What monitoring watches, which problems deserve to wake a person, how an average hides a slow tail, and who notices when the monitor itself goes quiet.","formats":{"gemtext":"https://engineer.company/glossary/monitoring/index.gmi","gopher":"https://engineer.company/glossary/monitoring/gophermap","html":"https://engineer.company/glossary/monitoring/","json":"https://engineer.company/glossary/monitoring/index.json","markdown":"https://engineer.company/glossary/monitoring/index.md","pdf":"https://engineer.company/glossary/monitoring/index.pdf"},"kind":"page","language":"en","title":"What is monitoring?","url":"https://engineer.company/glossary/monitoring/","version":1}