# Prometheus ## What it is Prometheus is an open-source monitoring system with its own time-series database at the centre: every few seconds it asks the programs it watches for their current numbers over HTTP, stores each value with a timestamp, and lets you query, graph and alert on what it has kept. The name covers the server on its own and, just as often, the small family of components that has grown up around it. It was built at SoundCloud, where work on it started in 2012, and it now belongs to the Cloud Native Computing Foundation. It joined in 2016 as the foundation's second hosted project, after Kubernetes, and graduated in 2018. Prometheus 3.0, announced in November 2024, was its first major release in seven years. The code is written mostly in Go and published under the Apache 2.0 licence. Everything it stores is a time series, named by a metric and a set of labels: key and value pairs that say which instance of a thing was measured, as in `api_http_requests_total{method="POST", handler="/messages"}`. Each sample in that series is a 64-bit floating-point value, or a native histogram, with a timestamp in milliseconds. A different value for any label is a different series. That is what makes the data easy to slice by host, route or status code, and also what makes it easy to overload. Programs expose their numbers on an HTTP endpoint, usually `/metrics`, through a client library compiled into the code or through an exporter: a small adapter that translates a system Prometheus cannot instrument directly, such as a Linux host or a database. The server finds its targets through service discovery or a static list and scrapes them on a schedule. A batch job that ends before the next scrape can push to a separate push gateway instead. Alerting rules run on the server, and firing alerts go to Alertmanager, which deduplicates and groups them, applies silences and routes them to email, PagerDuty, OpsGenie and similar receivers. It is usually weighed against push-based collectors in the Graphite and StatsD tradition, against general time-series databases such as InfluxDB, and against hosted metrics services. Grafana is the usual face in front of it. It stores numbers rather than events, so a log line or a trace belongs somewhere else, and the project is plain that it favours reliability over completeness: per-request billing, where every count has to be exact, is a job for a different system. ## Why it matters The decision Prometheus sits in is whether to run your own metrics database or rent one, and if you run it, how large it has to be. Running it carries no licence fee. The cost is a machine and the hours of whoever keeps it healthy. A hosted service takes that work away and replaces it with a charge that grows with the data sent to it, which turns every new label into a line on an invoice. For a few dozen machines the self-run server is usually the cheaper and simpler path. The sum shifts when the series count runs into the millions and the team would rather pay than staff it. The sizing arithmetic is kinder than most people expect. The project's storage documentation puts the average at one to two bytes per sample. A thousand series scraped every 15 seconds and kept for the default retention of 15 days come to 86.4 million samples, roughly 86–173 MB on disk. Disk is seldom what runs out. Memory is: the block currently being filled lives in memory, guarded against a crash by a write-ahead log, and every active series costs something there whether or not anyone ever queries it. Each server is deliberately self-contained. It leans on no network storage, so it keeps answering while the infrastructure it watches is falling over, which is the moment it is needed most. The price is that local storage is neither clustered nor replicated. High availability means two identical servers scraping the same targets, and history longer than a few weeks means sending a copy with `remote_write` to a store built to keep it. Pulling rather than receiving has a practical payoff too. Because the server does the asking, it notices when a target stops answering and records that as `up` dropping to zero. And an engineer can open the same `/metrics` page in a browser to see exactly what the server sees, which makes a disagreement between the graph and the application quick to settle. ## What it looks like in practice In this company's work Prometheus turns up as the collector behind a dashboard: once on a fleet of database servers, and once as the second version of a view the whole office watched. The database case was a fleet of [ten Ubuntu 20.04 LTS virtual servers carrying PostgreSQL 14 and MS SQL Server 2019](/portfolio/designed-deployed-and-maintained-10-postgresql-and-ms-13/). Prometheus and Grafana tracked CPU, memory, disk I/O and query performance as they happened, alongside automated backups, firewall rules and custom scripts for log analysis. Across all ten servers, uptime held at 99.9%. Database hosts suit scraping well, because the numbers that predict trouble there, a disk filling or a query slowing, move over hours, and a sample every few seconds sees them long before a user does. The office view shows the step from a home-made collector to Prometheus. [The company's first observability dashboard](/portfolio/developed-and-launched-the-company-s-first-observability-9/) began by reading CPU, RAM, disk, temperature and more from remote Linux servers over SSH, and watched Docker containers the same way. A second version was rebuilt on Prometheus and Grafana, around the metrics chosen with the DevOps and engineering teams: CPU utilisation, memory usage, service uptime and API latency, with alerts on threshold breaches. With the dashboard in place, teams detected and resolved incidents 40% faster. Polling over SSH is a fine start for a handful of machines; a scraper with labels and a query language is what lets the same view keep growing. ## Where it goes wrong **Memory climbs after a release that changed nothing important.** Someone added a label whose values are unbounded: a user ID, an email address, a full URL path, a request ID. Each new value became a new series, and the server now holds millions where it held thousands. The naming guide warns against exactly this. Keep label values to small, known sets, send per-request detail to logs or traces, and find the worst offenders with a query like this one: ```promql topk(10, count by (__name__) ({__name__=~".+"})) ``` **A graph shows a cliff or a spike that never happened.** A counter only rises, and it falls to zero when the process restarts. Plotting the raw value, or subtracting by hand, turns every restart into a crash or a surge. `rate()` understands resets, so take the rate first and add up afterwards, over a window several scrape intervals wide: ```promql sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) ``` A recording rule that sums a counter before the rate is taken hides the resets inside the sum and brings the false cliffs back. **A service goes down and its error alert stays green.** A rule such as an error rate above a threshold is evaluated over the series that exist. When the target disappears, so do its series, the expression returns an empty result, and the alert resolves itself at the worst possible moment. Alert on `up == 0` for every scrape target, use `absent()` for any series that must always be there, and have something outside the server check that the server itself is running. **Last quarter's numbers are gone when someone asks for them.** Local storage keeps 15 days by default and is a single copy on a single disk. It works as a window on recent behaviour, and as an archive it lets people down. Raise the retention with care, send what has to last to long-term storage through `remote_write`, and keep figures that money depends on in a database of record. ## Further reading - [Prometheus overview](https://prometheus.io/docs/introduction/overview/) — the project's own account of what it is, its components, and where it is the wrong fit. - [Data model](https://prometheus.io/docs/concepts/data_model/) — metric names, labels and samples, and why each label value starts a new series. - [Metric types](https://prometheus.io/docs/concepts/metric_types/) — counter, gauge, histogram and summary, and which questions each one can answer. - [Querying basics](https://prometheus.io/docs/prometheus/latest/querying/basics/) — PromQL selectors, range vectors and the types an expression can return. - [Storage](https://prometheus.io/docs/prometheus/latest/storage/) — two-hour blocks, the write-ahead log, retention and the remote-write interface. - [Metric and label naming](https://prometheus.io/docs/practices/naming/) — base units, the `_total` suffix and the warning about high-cardinality labels.