{"canonical_url":"https://engineer.company/glossary/metrics/","content_markdown":"# Metrics\n\n## What it is\n\nMetrics are measurements of a running service, each taken at a moment and kept\nas a value with a timestamp and a few labels, so that the same quantity can be\nread again a minute, a day or a month later. One named quantity is a metric;\nstored as a sequence of values over time it becomes a time series; and\nOpenTelemetry counts metrics as one of its signals, beside traces and logs.\nRequests served, memory in use and jobs waiting in a queue are typical metrics.\n\nWhat sets metrics apart from the other signals is aggregation. A log keeps one\nline per event, and a trace follows one request through the system. A metric\nfolds many measurements into a few statistics over a window of time: a count, a\nsum, a current value, a spread across buckets. The OpenTelemetry documentation\ncalls the result statistical information in aggregate, and that is both the\nstrength and the limit: a metric says how many requests failed in the last\nminute, and a log says which ones.\n\nOpenTelemetry and Prometheus agree on a small set of shapes, and choosing the\nright one is most of the design:\n\n- A **counter** accumulates and only goes up, like the odometer in a car:\n  requests served, bytes sent, errors seen. Prometheus allows one exception, a\n  reset to zero when the process restarts.\n- An **up-down counter**, in OpenTelemetry's terms, accumulates but can also go\n  down, such as the length of a queue as work arrives and leaves.\n- A **gauge** is a current value read at the moment it is read, like the fuel\n  gauge in the same car: memory in use, a temperature, requests in flight.\n- A **histogram** sorts measurements into buckets and keeps their count and\n  sum, so it can answer how many requests took less than a second.\n\nPrometheus also has a summary, which computes quantiles inside the program. The\nOpenTelemetry data model keeps it only as a legacy type, because summaries\ncannot always be merged meaningfully.\n\nThe text format that Prometheus scrapes shows the pieces together: a help line,\na type line, then one line per label combination with the name, the labels, the\nvalue and an optional timestamp in milliseconds since the epoch.\n\n```text\n# HELP http_requests_total The total number of HTTP requests.\n# TYPE http_requests_total counter\nhttp_requests_total{method=\"post\",code=\"200\"} 1027 1395066363000\nhttp_requests_total{method=\"post\",code=\"400\"} 3 1395066363000\n```\n\n## Why it matters\n\nMetrics are cheap enough to keep for everything. The memory they need grows\nwith the number of distinct label combinations, not with traffic; the\nOpenTelemetry documentation says so in as many words, in contrast with logs. A\ncounter incremented a billion times is still one number. That is what lets a\nteam count every request rather than a sample of them, and keep the counts long\nenough to compare this month with the last.\n\nThey turn arguments into measurements. Whether a host is big enough, whether a\nrelease made answers slower, how soon a disk fills: each has an answer in a\nseries recorded before anyone asked. Alerts are built on the same numbers, since\nan alert rule is a condition over a metric that someone decided was worth waking\nfor.\n\nThey also outlive the code that emits them, so the shape is worth getting\nright. The Prometheus naming guide asks for one unit and one quantity per\nmetric, in base units such as seconds and bytes rather than milliseconds or\nmegabytes, with the unit as a plural suffix and `_total` on an accumulating\ncount. Names such as `http_request_duration_seconds` and\n`process_cpu_seconds_total` then say on their face what they hold.\n\nHow totals travel matters as well. Prometheus reports cumulative totals, so\ngaps from an intermittent collection failure average out, at the price of\nmemory in the sender in proportion to its label combinations. StatsD reports\ndeltas, the change since the last report, which moves that cost out of the\nprocess.\n\n## What it looks like in practice\n\nThe company's own public-services host is a single-core machine with 512 MB of\nmemory, of which about 464 MB is usable, and on it memory became the argument\nthat settled decisions. Reporting there is a pull: seven commands that read the\nhost and render Markdown, change nothing and run only when asked. When the\nquestion came up whether\n[that one 512 MB host could also carry an onion service](/portfolio/a-whole-company-on-one-small-server/),\nthe answer came from a recorded day of measured samples rather than an opinion:\navailable memory never fell below about 310 MB of 464, swap sat at 2.6 per cent\nand the processor was 99.7 per cent idle.\n\nThose reports later had to say what their numbers were worth.\n[Six read-only collectors were made to state what they never read](/portfolio/made-six-collectors-state-what-they-skipped/)\nonce it was clear that a blind collection and a healthy host rendered the same\npage: a mirror whose daemon changed its log format produced the same zero as a\nmirror nobody visited. Thresholds were fitted to the 116 samples the metrics\nreport had already measured, with the observed ranges recorded beside the\nbounds, and the report tells three outcomes apart: a breach, no breach, and not\njudged because nothing measured it. Each collector reports its runtime and its\nblindness, such as files read against files unreadable, even when the answer is\nzero. The bounds are daily averages, which can hide a one-minute spike. The\nfifteen-minute switch checks available memory and free disk at each run, but a\nbrief breach between its checks can still go unseen.\n\nThe most useful metrics often come from the problem rather than the host. The\nnote on\n[selling limited stock without overselling](/notes/selling-limited-stock-without-overselling/)\nputs lock waits, expired holds and the distance between the gate's count and\nthe database's on one dashboard, and runs the invariant\n`available + held + sold = total` on a schedule, charted and paged on at any\ndrift. Each of those numbers comes from knowing how that sale goes wrong.\n\n## Where it goes wrong\n\nMetrics rarely break loudly. The graph keeps drawing, and the number on it\nquietly means something other than what its name says.\n\n**A value that falls is published as a counter.** The Prometheus metric types\npage says not to use a counter for anything that can decrease, such as the\nnumber of running processes: that is a gauge. The query functions treat any\nfall in a counter as a restart and correct for it, so a value that falls on\npurpose shows up as increases that never happened.\n\n**Units drift between services.** One service reports a duration in\nmilliseconds, another in seconds, and a panel that compares them is off by a\nfactor of a thousand. The naming guide says a metric must refer to a single\nunit and should use base units. Converting once, at the source, is cheaper than\nremembering the difference in every query.\n\n**A sum across labels means nothing.** The naming guide's rule of thumb is that\nthe sum or the average over all of a metric's labels should be meaningful. Its\nown example: the capacities of several queues belong together in one metric,\nwhile a queue's capacity and its current length do not.\n\n**Unmeasured reads as zero.** A collector that has stopped reading its input\nleaves a zero, or a missing row, that looks exactly like a quiet system. The\nOpenTelemetry data model carries a flag for a point with no recorded value, its\ncounterpart to the Prometheus staleness marker, because missing data and a\nvalue of zero are different facts. Report them differently, and give \"not\njudged\" its own outcome, as the collectors above do.\n\n**The window hides the spike.** A gauge is a sample at a moment, and the\nOpenTelemetry data model reports only the last value in each interval, so a\nburst between two samples leaves no trace. A daily average smooths a minute at\nthe ceiling away in the same way. Choose the window from the failure that has\nto be seen, and keep a fast check beside a slow one when both matter.\n\n**Overflow undercounts a filtered query.** The OpenTelemetry SDK tracks at most\n2,000 distinct attribute combinations per metric stream by default, and folds\nany further ones into a single data point marked `otel.metric.overflow=true`.\nThe total stays correct, but overflow replaces the whole combination, not just\nits noisy part. In the documentation's own example, a request counter carries\n`url.path` and `success`; once it overflows, a query for `success=false`\nundercounts, and an error-rate alert built on it can stop firing. Keep\nattributes to small, known sets, and watch for the overflow marker.\n\n## Further reading\n\n- [OpenTelemetry: Metrics](https://opentelemetry.io/docs/concepts/signals/metrics/)\n  — instrument kinds, aggregation and the cardinality limit.\n- [OpenTelemetry: Metrics data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/)\n  — gauges, sums, histograms, temporality and the no-recorded-value flag.\n- [Prometheus: Metric types](https://prometheus.io/docs/concepts/metric_types/)\n  — counter, gauge, histogram and summary.\n- [Prometheus: Metric and label naming](https://prometheus.io/docs/practices/naming/)\n  — base units, unit suffixes and `_total`.\n- [Prometheus: Histograms and summaries](https://prometheus.io/docs/practices/histograms/)\n  — where quantiles are computed, and which ones can be combined.\n- [Prometheus: Exposition formats](https://prometheus.io/docs/instrumenting/exposition_formats/)\n  — the text format line by line.\n\n\u003chttps://engineer.company/glossary/metrics/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"How counters, gauges and histograms turn a running service into numbers you can trust, how to name them, and how a metric quietly stops telling the truth.","formats":{"gemtext":"https://engineer.company/glossary/metrics/index.gmi","gopher":"https://engineer.company/glossary/metrics/gophermap","html":"https://engineer.company/glossary/metrics/","json":"https://engineer.company/glossary/metrics/index.json","markdown":"https://engineer.company/glossary/metrics/index.md","pdf":"https://engineer.company/glossary/metrics/index.pdf"},"kind":"page","language":"en","title":"What are metrics?","url":"https://engineer.company/glossary/metrics/","version":1}