{"canonical_url":"https://engineer.company/glossary/prometheus/","content_markdown":"# Prometheus\n\n## What it is\n\nPrometheus is an open-source monitoring system with its own time-series database\nat the centre: every few seconds it asks the programs it watches for their\ncurrent numbers over HTTP, stores each value with a timestamp, and lets you\nquery, graph and alert on what it has kept. The name covers the server on its\nown and, just as often, the small family of components that has grown up around\nit.\n\nIt was built at SoundCloud, where work on it started in 2012, and it now belongs\nto the Cloud Native Computing Foundation. It joined in 2016 as the foundation's\nsecond hosted project, after Kubernetes, and graduated in 2018. Prometheus 3.0,\nannounced in November 2024, was its first major release in seven years. The\ncode is written mostly in Go and published under the Apache 2.0 licence.\n\nEverything it stores is a time series, named by a metric and a set of labels:\nkey and value pairs that say which instance of a thing was measured, as in\n`api_http_requests_total{method=\"POST\", handler=\"/messages\"}`. Each sample in\nthat series is a 64-bit floating-point value, or a native histogram, with a\ntimestamp in milliseconds. A different value for any label is a different\nseries. That is what makes the data easy to slice by host, route or status code,\nand also what makes it easy to overload.\n\nPrograms expose their numbers on an HTTP endpoint, usually `/metrics`, through a\nclient library compiled into the code or through an exporter: a small adapter\nthat translates a system Prometheus cannot instrument directly, such as a Linux\nhost or a database. The server finds its targets through service discovery or a\nstatic list and scrapes them on a schedule. A batch job that ends before the\nnext scrape can push to a separate push gateway instead. Alerting rules run on\nthe server, and firing alerts go to Alertmanager, which deduplicates and groups\nthem, applies silences and routes them to email, PagerDuty, OpsGenie and similar\nreceivers.\n\nIt is usually weighed against push-based collectors in the Graphite and StatsD\ntradition, against general time-series databases such as InfluxDB, and against\nhosted metrics services. Grafana is the usual face in front of it. It stores\nnumbers rather than events, so a log line or a trace belongs somewhere else, and\nthe project is plain that it favours reliability over completeness: per-request\nbilling, where every count has to be exact, is a job for a different system.\n\n## Why it matters\n\nThe decision Prometheus sits in is whether to run your own metrics database or\nrent one, and if you run it, how large it has to be.\n\nRunning it carries no licence fee. The cost is a machine and the hours of\nwhoever keeps it healthy. A hosted service takes that work away and replaces it\nwith a charge that grows with the data sent to it, which turns every new label\ninto a line on an invoice. For a few dozen machines the self-run server is\nusually the cheaper and simpler path. The sum shifts when the series count runs\ninto the millions and the team would rather pay than staff it.\n\nThe sizing arithmetic is kinder than most people expect. The project's storage\ndocumentation puts the average at one to two bytes per sample. A thousand series\nscraped every 15 seconds and kept for the default retention of 15 days come to\n86.4 million samples, roughly 86–173 MB on disk. Disk is seldom what runs out.\nMemory is: the block currently being filled lives in memory, guarded against a\ncrash by a write-ahead log, and every active series costs something there\nwhether or not anyone ever queries it.\n\nEach server is deliberately self-contained. It leans on no network storage, so\nit keeps answering while the infrastructure it watches is falling over, which is\nthe moment it is needed most. The price is that local storage is neither\nclustered nor replicated. High availability means two identical servers\nscraping the same targets, and history longer than a few weeks means sending a\ncopy with `remote_write` to a store built to keep it.\n\nPulling rather than receiving has a practical payoff too. Because the server\ndoes the asking, it notices when a target stops answering and records that as\n`up` dropping to zero. And an engineer can open the same `/metrics` page in a\nbrowser to see exactly what the server sees, which makes a disagreement between\nthe graph and the application quick to settle.\n\n## What it looks like in practice\n\nIn the founder's work Prometheus turns up as the collector behind a dashboard:\nonce on a fleet of database servers, and once as the second version of a view\nthe whole office watched.\n\nThe database case was a fleet of\n[ten Ubuntu 20.04 LTS virtual servers carrying PostgreSQL 14 and MS SQL Server 2019](/portfolio/ran-10-postgresql-and-ms-sql-servers-on-ubuntu/).\nPrometheus and Grafana tracked CPU, memory,\ndisk I/O and query performance as they happened, alongside automated backups,\nfirewall rules and custom scripts for log analysis. Across all ten servers,\nuptime held at 99.9%. Database hosts suit scraping well, because the numbers\nthat predict trouble there, a disk filling or a query slowing, move over hours,\nand a sample every few seconds sees them long before a user does.\n\nThe office view shows the step from a home-made collector to Prometheus.\nIn an earlier role, the founder built\n[an organisation's first observability dashboard](/portfolio/built-the-companys-first-observability-dashboard/).\nIt began by reading CPU, RAM, disk, temperature and more from remote Linux servers\nover SSH, and watched Docker containers the same way. A second version was\nrebuilt on Prometheus and Grafana, around the metrics chosen with the DevOps\nand engineering teams: CPU utilisation, memory usage, service uptime and API\nlatency, with alerts on threshold breaches. With the dashboard in place, teams\ndetected and resolved incidents 40% faster. Polling over SSH is a fine start\nfor a handful of machines; a scraper with labels and a query language is what\nlets the same view keep growing.\n\n## Where it goes wrong\n\n**Memory climbs after a release that changed nothing important.** Someone added\na label whose values are unbounded: a user ID, an email address, a full URL\npath, a request ID. Each new value became a new series, and the server now holds\nmillions where it held thousands. The naming guide warns against exactly this.\nKeep label values to small, known sets, send per-request detail to logs or\ntraces, and find the worst offenders with a query like this one:\n\n```promql\ntopk(10, count by (__name__) ({__name__=~\".+\"}))\n```\n\n**A graph shows a cliff or a spike that never happened.** A counter only rises,\nand it falls to zero when the process restarts. Plotting the raw value, or\nsubtracting by hand, turns every restart into a crash or a surge. `rate()`\nunderstands resets, so take the rate first and add up afterwards, over a window\nseveral scrape intervals wide:\n\n```promql\nsum by (job) (rate(http_requests_total{status=~\"5..\"}[5m]))\n```\n\nA recording rule that sums a counter before the rate is taken hides the resets\ninside the sum and brings the false cliffs back.\n\n**A service goes down and its error alert stays green.** A rule such as an error\nrate above a threshold is evaluated over the series that exist. When the target\ndisappears, so do its series, the expression returns an empty result, and the\nalert resolves itself at the worst possible moment. Alert on `up == 0` for\nevery scrape target, use `absent()` for any series that must always be there,\nand have something outside the server check that the server itself is running.\n\n**Last quarter's numbers are gone when someone asks for them.** Local storage\nkeeps 15 days by default and is a single copy on a single disk. It works as a\nwindow on recent behaviour, and as an archive it lets people down. Raise the\nretention with care, send what has to last to long-term storage through\n`remote_write`, and keep figures that money depends on in a database of record.\n\n## Further reading\n\n- [Prometheus overview](https://prometheus.io/docs/introduction/overview/)\n  — the project's own account of what it is, its components, and where it is\n  the wrong fit.\n- [Data model](https://prometheus.io/docs/concepts/data_model/) — metric names,\n  labels and samples, and why each label value starts a new series.\n- [Metric types](https://prometheus.io/docs/concepts/metric_types/) — counter,\n  gauge, histogram and summary, and which questions each one can answer.\n- [Querying basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)\n  — PromQL selectors, range vectors and the types an expression can return.\n- [Storage](https://prometheus.io/docs/prometheus/latest/storage/) — two-hour\n  blocks, the write-ahead log, retention and the remote-write interface.\n- [Metric and label naming](https://prometheus.io/docs/practices/naming/) — base\n  units, the `_total` suffix and the warning about high-cardinality labels.\n\n\u003chttps://engineer.company/glossary/prometheus/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"How Prometheus pulls and stores metrics, how far one server goes, what a careless label costs, and the alerts that stay quiet when they should fire.","formats":{"gemtext":"https://engineer.company/glossary/prometheus/index.gmi","gopher":"https://engineer.company/glossary/prometheus/gophermap","html":"https://engineer.company/glossary/prometheus/","json":"https://engineer.company/glossary/prometheus/index.json","markdown":"https://engineer.company/glossary/prometheus/index.md","pdf":"https://engineer.company/glossary/prometheus/index.pdf"},"kind":"page","language":"en","title":"What is Prometheus?","url":"https://engineer.company/glossary/prometheus/","version":1}