{"canonical_url":"https://engineer.company/glossary/alert/","content_markdown":"# Alert\n\n## What it is\n\nAn alert is an automatic signal that some watched condition has stayed true long enough to be worth\ninterrupting somebody. Some teams call it an alarm, and an alert loud enough to wake the person on\ncall is usually called a page.\n\nThe word covers two different objects, and Prometheus keeps them apart in a way that makes the\ndistinction easy to see. An alerting rule in Prometheus produces alerts: one per distinct set of\nlabels for which the expression returns a result. Those alerts go to a separate program, the\nAlertmanager, which turns them into notifications: an email, a chat message, a call to an on-call\nservice. The project says plainly that alerting rules on their own are not a full notification\nsystem. The split matters because the numbers diverge. Fifty alerts can arrive as one message, one\nalert can produce a reminder every few hours, and a silenced alert reaches nobody at all while still\nfiring.\n\nAn alert has a short life cycle. While the expression returns nothing, the alert is inactive. When\nit first returns a result, the alert is pending, and it stays pending for as long as the rule's\n`for` clause asks. If the condition is still true at the end of that wait, the alert fires and is\nhanded on for delivery. When the condition stops being true, the alert resolves. A newer clause,\n`keep_firing_for`, holds a firing alert open for a while after its condition clears, which the\ndocumentation offers as a guard against flapping and false resolutions. Prometheus also records the\npending and firing states as a time series named `ALERTS`, so the alerts themselves can be queried\nand graphed.\n\nA rule that waits twenty minutes before firing, and ten more before it lets go, looks like this:\n\n```yaml\ngroups:\n    - name: exports\n      rules:\n          - alert: ExportQueueStalled\n            expr: exports_waiting \u003e 0 and rate(exports_done_total[15m]) == 0\n            for: 20m\n            keep_firing_for: 10m\n            labels:\n                severity: page\n                team: billing\n            annotations:\n                summary: \"{{ $value }} exports queued and none finishing\"\n```\n\nLabels and annotations look alike and do different jobs. Labels are part of what makes an alert this\nalert and not another one, and the Alertmanager reads them to decide where it goes: `severity: page`\nmight reach a phone while `severity: ticket` lands in a queue. Annotations carry the words a person\nneeds once the message arrives: a one-line summary, a longer description, a link to the runbook that\nsays what to do first. Annotations can be templated with the alert's labels and the value that\ntriggered it, which is how the summary above can quote a number.\n\nAlert and incident are separate words for a reason. An alert reports that a condition held; deciding\nthat the condition amounts to an incident is a person's call.\n\n## Why it matters\n\nAn alert is the one piece of monitoring that does not wait to be looked at. A metric sits in\nstorage, a dashboard waits on a screen, a log waits for a search, and an alert reaches into\nsomeone's evening. That makes every alerting rule a small standing claim on a person's attention,\nand the claim is only fair if the person can do something with it. The Prometheus guidance on\nalerting puts it as advice to avoid pages where there is nothing to do, and to keep alerting simple.\n\nThe second reason is arithmetic. A single cause usually trips many rules at once. When a network\npartition cuts half of a service's instances off from their database, each instance can fire its own\nalert, and the Alertmanager documentation uses exactly that case to show hundreds of alerts becoming\na single notification. The Alertmanager collects alerts that share the labels named in `group_by`,\nwaits `group_wait` (30 seconds by default) so the first message holds the whole batch, sends updates\nfor the group no more often than `group_interval` (5 minutes), and repeats a still-firing message\nafter `repeat_interval` (4 hours). Inhibition goes a step further and suppresses some alerts while\nanother is firing, so an alert saying a whole cluster is unreachable can quiet the per-service\nalerts from inside it. Silences mute alerts that match a set of labels for a fixed period, typically\nduring planned maintenance.\n\nThe third reason is the end of the story. Most receivers send a resolved message when the alert\nclears, but the email receiver's `send_resolved` setting defaults to false, so a team that alerts by\nemail can see the problem announced and never see it end.\n\n## What it looks like in practice\n\nThe company's own host uses a missing-signal alarm. Its 464 MB of memory serves a web server and a forge, and memory exhaustion and a full disk are two important failure risks. [A timer on the host pings an outside service only while its available-memory and free-disk thresholds pass](/portfolio/built-dead-mans-switch-host-monitoring/), every fifteen minutes with some jitter, and the outside service raises the alarm when the expected pings stop. Failures that prevent the script from running or reaching that service, including a lost network or a failed boot, produce a missing ping. A received signal confirms that the script ran and its available-memory and free-disk thresholds passed; it does not certify general host health or that the site is serving. Monitoring can expose warning signs, although the timing depends on the workload and the conditions observed.\n\nThe same inversion can guard scheduled work. The\n[backup implementation separates success signals for backup and restore](/portfolio/backups-that-have-actually-been-restored/).\nIts scripts and systemd units support encrypted off-host snapshots, integrity checks and a\nmonthly known-file restore test; a success signal is sent only after its workflow succeeds.\nThat is the design of the alert path, not proof that it was active. The implementation record\ndated 2026-10-27 says the timers were disabled without a configured repository and records no\ncompleted live restore. Silence can raise an external alert once the workflow and its monitor\nare activated; configuration alone does not establish that they are running.\n\nInverted alerts have a failure mode of their own, and it showed up when the platform was about to\ngrow from one host to two. Sharing one switch between both hosts would have\n[weakened the alerting it was meant to extend](/portfolio/split-every-operational-secret-per-host/):\nas long as either host keeps pinging, the check stays green while the other one is dead. Each switch\nbecame a per-host value, and the existing host's switch entries were left blank on purpose, so the\nconfiguration says the wiring exists and the value still has to be supplied.\n\n## Where it goes wrong\n\n**A measured value ends up in a label.** Because labels identify an alert, a label that carries the\ncurrent number turns each change of that number into a new alert and resolves the previous one. The\nreceiver sees a stream of fire-and-resolve pairs for one problem. Keep labels for identity and\nrouting, and put numbers in annotations.\n\n**The alert flaps.** A condition that hovers around its threshold fires, resolves and fires again\nevery few minutes, and each resolved message tells the reader something false. Lengthen `for` so a\nblip clears before anyone hears of it, and use `keep_firing_for` so a brief dip below the threshold\ndoes not end the alert.\n\n**A silence outlives its reason.** Someone mutes an alert for maintenance with a broad matcher and a\ngenerous end time, and next week's real failure arrives muted. Match silences as narrowly as the\nwork allows, give them an end time close to when the work should finish, and write who set them and\nwhy in the comment field the Alertmanager provides.\n\n**A broad route swallows the specific ones.** The Alertmanager walks the routing tree in order and,\nunless a route sets `continue: true`, stops at the first child that matches. A catch-all route\nplaced above a team's own route takes that team's alerts with it. Put the narrowest routes first and\ntest the tree with sample labels before relying on it.\n\n**Everything is urgent.** Disk at 70%, a certificate expiring next month and a payment service\nreturning errors all page the same phone, and the phone soon gets ignored. Page only for conditions\na person has to act on now, send the rest to a ticket queue or a daily summary, and delete a rule\nthat has fired for months without anyone acting on it.\n\n## Further reading\n\n- [Alerting overview](https://prometheus.io/docs/alerting/latest/overview/) — how Prometheus\n  alerting rules and the Alertmanager divide the work, and the steps to set the two up.\n- [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) — the\n  rule syntax, including `for`, `keep_firing_for`, labels, annotations and templating.\n- [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) — grouping, inhibition,\n  silences and running more than one Alertmanager for high availability.\n- [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/) — the\n  routing tree, the grouping timers, inhibit rules and each receiver's settings, `send_resolved`\n  among them.\n- [Alerting practices](https://prometheus.io/docs/practices/alerting/) — the project's own advice on\n  what deserves a page and what belongs elsewhere.\n\n\u003chttps://engineer.company/glossary/alert/\u003e\n","date_modified":"2026-11-12T05:34:23Z","description":"How an alert moves from pending to firing to resolved, what its labels and annotations each do, and why the most dependable alarm on a small host is a silence.","formats":{"gemtext":"https://engineer.company/glossary/alert/index.gmi","gopher":"https://engineer.company/glossary/alert/gophermap","html":"https://engineer.company/glossary/alert/","json":"https://engineer.company/glossary/alert/index.json","markdown":"https://engineer.company/glossary/alert/index.md","pdf":"https://engineer.company/glossary/alert/index.pdf"},"kind":"page","language":"en","title":"What is an alert?","url":"https://engineer.company/glossary/alert/","version":1}