# Logging ## What it is Logging, also called event logging, is a program keeping a running account of its own behaviour, one entry per event: a request served, a password refused, a retry, a crash. Each entry, a log line or log record, carries at least a time and a message, so that someone can later reconstruct what happened and in what order. The OpenTelemetry project defines a log as a timestamped text record, structured or unstructured, with optional metadata, and notes that of all the telemetry signals, logs have the biggest legacy. That legacy has a standard. Syslog, the long-standing protocol for passing event messages to a collector, was standardised as RFC 5424 in March 2009, which obsoletes the older RFC 3164, and it gives every message one of eight severities, from 0 for Emergency to 7 for Debug. Most logging libraries carry a shorter ladder of the same idea: debug, info, warning, error. OpenTelemetry's log data model maps them all onto one number from 1 to 24, in six bands from TRACE to FATAL. An unstructured line is free text: easy for a person to read, and something a machine can only take apart with a regular expression written for that exact wording. A structured line follows a fixed layout, so every entry has the same named fields with the same types and meanings. OpenTelemetry prefers structured logs in production and adds a warning that is easy to miss: a line encoded as JSON is not structured just because it parses; what makes it structured is a stable layout. Shown across several lines here, a structured entry is a single line on disk: ```json { "time": "2026-09-14T03:12:07.412Z", "severity": "WARN", "service": "checkout", "event": "payment.retry", "attempt": 3, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736" } ``` Metrics and traces are the signals usually collected beside it. Metrics are numbers sampled over time, cheap to keep and good at saying how often. Traces follow one request through every service it touches. A log keeps the detail of single events, which makes it what someone reads after an alert to find out why. OpenTelemetry ties logs to traces by attaching the IDs of the active trace and span to each log record. ## Why it matters A log is the only account of the past that most systems leave behind. A metric can say that errors rose at 03:12; the log says which request, from which service, with which value, and what the program did next. When something breaks at night and has recovered by morning, the log is the witness still there. That account has a price, paid three ways. Every line costs disk, input and output, and processor time, and more again if it is shipped to a central store, where a hosted service typically bills by the volume taken in. Every line is also data held about someone: a web server's access log holds the addresses and browsers of everyone who called. And every line has to be guarded. The OWASP Logging Cheat Sheet asks that each event record when, where, who and what, and that the logs themselves be protected against tampering and against unauthorised access, modification and deletion, because they may contain personal and other sensitive information. At the scale of an organisation, NIST Special Publication 800-92, the Guide to Computer Security Log Management by Karen Kent and Murugiah Souppaya, published in September 2006, treats log management as infrastructure and processes run across the whole enterprise rather than a setting in one program. Logs also work best next to numbers. The Prometheus project's instrumentation guide suggests a counter for every line of logging code, so that when an interesting message turns up, it is possible to tell how often it has occurred and over how long a period, and suggests comparing the totals of info, warning and error lines from one release to the next. ## What it looks like in practice The site's own access log is described line by line on the [privacy page](/privacy/). Each request records the page and time, the response status, the browser's self-description, the name of the referring site and a blurred network address: the last part is replaced with zeroes at the moment the line is written, before anything touches the disk. A line lives about thirty days. Files rotate at about 10 MB and a rotated file is deleted once fourteen newer ones exist or it turns thirty days old, and that rule had a gap: the quiet addresses filled 10 MB so slowly that their files had never rotated and still held every line since logging began. A daily job now closes any file whose oldest line is more than a day old and deletes closed files after thirty days. Four addresses were in the old state, and the page says so. One short session of a macOS application wrote 610,996 log lines. Each event it handled produced lines of its own, so the volume of logging was proportional to the volume of whatever was going wrong, and that logging was one of three causes of [memory filling at roughly 41 MB a second](/portfolio/stopped-an-application-filling-memory-at-41mb-a-second-155/), which in the recorded incident left 111 GB of pages compressed on a 36 GB machine. With bounded event buffers, subscription by event type and a logging budget that collapses repeats, the same session now writes 1,411 lines. The budget discards information by design: when something goes wrong quickly now, the record of it is deliberately incomplete. On a production host, a [hardening pass over thirteen systemd units](/portfolio/cut-systemd-sandbox-exposure-across-every-unit-119/) turned up one directive misspelled in three unit templates for as long as those roles had existed. The service manager answers a key it does not recognise by logging that fact and starting the unit anyway, so the only trace of the mistake was a warning in a log; it came to light while the units were being rewritten. Now every unit goes through the service manager's own verifier during each converge, so the same typo fails a run. ## Where it goes wrong **The log grows fastest when the system is already failing.** A loop that logs each retry writes a handful of lines on a good day and thousands a second on a bad one, taking disk, memory and processor time from a system with none to spare. OWASP asks teams to verify that logging cannot be turned into a way to exhaust system resources, for example by filling the disk. Give every hot path a budget, collapse repeated messages into one line with a count, and keep a counter beside the message so the rate survives when lines are dropped. **Secrets and personal data get written down.** A request logged whole carries whatever was in it. OWASP's list of what should usually be removed, masked, hashed or encrypted before it reaches a log includes session identifiers, access tokens, passwords, database connection strings, encryption keys and payment card data. Do it when the line is written, as the access log above does with addresses, because by the time anyone cleans up, the line may sit in a backup or an archive. **Input from outside writes a fake line.** A user-supplied value containing a carriage return or newline, written unescaped into a text log, can start a new line that looks exactly like one the application wrote. OWASP calls this log injection and asks that such characters be sanitised and that data be encoded for the format it is written in. A structured logger that writes each value as an escaped JSON string does this by construction. **A log that goes quiet passes for a healthy system.** A full disk, a rotated file the program never reopened or a collector that went down all look the same from outside: the lines stop arriving. OWASP asks for a way to detect that logging has stopped. Count lines per minute for each source and alert when a count drops to zero, so a gap in the log becomes an alert of its own. ## Further reading - [Logs](https://opentelemetry.io/docs/concepts/signals/logs/) — OpenTelemetry on what a log is, why structured logs are preferred and how logs are tied to traces. - [Logs Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) — the fields of a log record, its two timestamps and the severity scale from 1 to 24. - [RFC 5424: The Syslog Protocol](https://www.rfc-editor.org/rfc/rfc5424) — the syslog message format and its eight severities. - [NIST SP 800-92](https://csrc.nist.gov/pubs/sp/800/92/final) — the Guide to Computer Security Log Management. - [Logging Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html) — OWASP on which events to record, which data to keep out and how to protect the result. - [Instrumentation](https://prometheus.io/docs/practices/instrumentation/) — the Prometheus project's practices for instrumenting code, logging included.