{"canonical_url":"https://engineer.company/glossary/logging/","content_markdown":"# Logging\n\n## What it is\n\nLogging, also called event logging, is a program keeping a running account of\nits own behaviour, one entry per event: a request served, a password refused, a\nretry, a crash. Each entry, a log line or log record, carries at least a time\nand a message, so that someone can later reconstruct what happened and in what\norder. The OpenTelemetry project defines a log as a timestamped text record,\nstructured or unstructured, with optional metadata, and notes that of all the\ntelemetry signals, logs have the biggest legacy.\n\nThat legacy has a standard. Syslog, the long-standing protocol for passing event\nmessages to a collector, was standardised as RFC 5424 in March 2009, which\nobsoletes the older RFC 3164, and it gives every message one of eight\nseverities, from 0 for Emergency to 7 for Debug. Most logging libraries carry a\nshorter ladder of the same idea: debug, info, warning, error. OpenTelemetry's\nlog data model maps them all onto one number from 1 to 24, in six bands from\nTRACE to FATAL.\n\nAn unstructured line is free text: easy for a person to read, and something a\nmachine can only take apart with a regular expression written for that exact\nwording. A structured line follows a fixed layout, so every entry has the same\nnamed fields with the same types and meanings. OpenTelemetry prefers structured\nlogs in production and adds a warning that is easy to miss: a line encoded as\nJSON is not structured just because it parses; what makes it structured is a\nstable layout. Shown across several lines here, a structured entry is a single\nline on disk:\n\n```json\n{\n    \"time\": \"2026-09-14T03:12:07.412Z\",\n    \"severity\": \"WARN\",\n    \"service\": \"checkout\",\n    \"event\": \"payment.retry\",\n    \"attempt\": 3,\n    \"trace_id\": \"4bf92f3577b34da6a3ce929d0e0e4736\"\n}\n```\n\nMetrics and traces are the signals usually collected beside it. Metrics are\nnumbers sampled over time, cheap to keep and good at saying how often. Traces\nfollow one request through every service it touches. A log keeps the detail of\nsingle events, which makes it what someone reads after an alert to find out why.\nOpenTelemetry ties logs to traces by attaching the IDs of the active trace and\nspan to each log record.\n\n## Why it matters\n\nA log is the only account of the past that most systems leave behind. A metric\ncan say that errors rose at 03:12; the log says which request, from which\nservice, with which value, and what the program did next. When something breaks\nat night and has recovered by morning, the log is the witness still there.\n\nThat account has a price, paid three ways. Every line costs disk, input and\noutput, and processor time, and more again if it is shipped to a central store,\nwhere a hosted service typically bills by the volume taken in. Every line is\nalso data held about someone: a web server's access log holds the addresses and\nbrowsers of everyone who called. And every line has to be guarded. The OWASP\nLogging Cheat Sheet asks that each event record when, where, who and what, and\nthat the logs themselves be protected against tampering and against unauthorised\naccess, modification and deletion, because they may contain personal and other\nsensitive information.\n\nAt the scale of an organisation, NIST Special Publication 800-92, the Guide to\nComputer Security Log Management by Karen Kent and Murugiah Souppaya, published\nin September 2006, treats log management as infrastructure and processes run\nacross the whole enterprise rather than a setting in one program.\n\nLogs also work best next to numbers. The Prometheus project's instrumentation\nguide suggests a counter for every line of logging code, so that when an\ninteresting message turns up, it is possible to tell how often it has occurred\nand over how long a period, and suggests comparing the totals of info, warning\nand error lines from one release to the next.\n\n## What it looks like in practice\n\nThe site's own access log is described line by line on the\n[privacy page](/privacy/). Each request records the page and time, the response\nstatus, the browser's self-description, the name of the referring site and a\nblurred network address: the last part is replaced with zeroes at the moment the\nline is written, before anything touches the disk. A line lives about thirty\ndays. The server software closes each file once a day, or sooner at about 10 MB,\nand deletes a closed file twenty-eight days later, and every night a check reads\nevery file back and fails on any line older than thirty days. It has gone wrong\ntwice, and the page records both. In September 2026 the quiet addresses filled\n10 MB so slowly that their files had never been closed and still held every line\nsince logging began, so a daily job was added to close them. On 26 October\n2026 seven of the thirteen addresses that keep a log were still writing into\nfiles that job had marked as closed, and one kept a closed file forty-seven days\nold. The job is gone, and the server software now closes and deletes the files\nitself.\n\nA full test run of a macOS application recorded 610,996 core log lines. A burst of core log\nevents was retained in an unbounded main stream and copied to subscriber streams whose\nconsumers did not use them. The\n[incident record](/portfolio/an-app-that-ate-41-mb-a-second-stopped/)\nseparately documents diagnostic reports of 111 GB of compressed pages on a 36 GB machine\nand a footprint sample showing growth of about 41 MB per second during a connection retry\nloop. Bounded buffers, subscriptions filtered by event type and a logging rate budget\naddressed that event path. The recorded full test-run comparison reduced core log lines to\n1,411. It is not a production benchmark or a measured post-fix memory figure. The budget\ndiscards information by design: repetitive diagnostic events can be dropped and affected\nstate refreshed instead of retaining unlimited events. Separate shutdown timeouts required\ntheir own correction.\n\nOn a production host, a\n[hardening pass over thirteen systemd units](/portfolio/cut-systemd-sandbox-exposure-on-every-unit/)\nturned up one directive misspelled in three unit templates for as long as those\nroles had existed. The service manager answers a key it does not recognise by\nlogging that fact and starting the unit anyway, so the only trace of the mistake\nwas a warning in a log; it came to light while the units were being rewritten.\nNow every unit goes through the service manager's own verifier during each\nconverge, so the same typo fails a run.\n\n## Where it goes wrong\n\n**The log grows fastest when the system is already failing.** A loop that logs\neach retry writes a handful of lines on a good day and thousands a second on a\nbad one, taking disk, memory and processor time from a system with none to\nspare. OWASP asks teams to verify that logging cannot be turned into a way to\nexhaust system resources, for example by filling the disk. Give every hot path a\nbudget, collapse repeated messages into one line with a count, and keep a\ncounter beside the message so the rate survives when lines are dropped.\n\n**Secrets and personal data get written down.** A request logged whole carries\nwhatever was in it. OWASP's list of what should usually be removed, masked,\nhashed or encrypted before it reaches a log includes session identifiers, access\ntokens, passwords, database connection strings, encryption keys and payment card\ndata. Do it when the line is written, as the access log above does with\naddresses, because by the time anyone cleans up, the line may sit in a backup or\nan archive.\n\n**Input from outside writes a fake line.** A user-supplied value containing a\ncarriage return or newline, written unescaped into a text log, can start a new\nline that looks exactly like one the application wrote. OWASP calls this log\ninjection and asks that such characters be sanitised and that data be encoded\nfor the format it is written in. A structured logger that writes each value as\nan escaped JSON string does this by construction.\n\n**A log that goes quiet passes for a healthy system.** A full disk, a rotated\nfile the program never reopened or a collector that went down all look the same\nfrom outside: the lines stop arriving. OWASP asks for a way to detect that\nlogging has stopped. Count lines per minute for each source and alert when a\ncount drops to zero, so a gap in the log becomes an alert of its own.\n\n## Further reading\n\n- [Logs](https://opentelemetry.io/docs/concepts/signals/logs/) — OpenTelemetry\n  on what a log is, why structured logs are preferred and how logs are tied to\n  traces.\n- [Logs Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) —\n  the fields of a log record, its two timestamps and the severity scale from 1\n  to 24.\n- [RFC 5424: The Syslog Protocol](https://www.rfc-editor.org/rfc/rfc5424) — the\n  syslog message format and its eight severities.\n- [NIST SP 800-92](https://csrc.nist.gov/pubs/sp/800/92/final) — the Guide to\n  Computer Security Log Management.\n- [Logging Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html)\n  — OWASP on which events to record, which data to keep out and how to protect\n  the result.\n- [Instrumentation](https://prometheus.io/docs/practices/instrumentation/) — the\n  Prometheus project's practices for instrumenting code, logging included.\n\n\u003chttps://engineer.company/glossary/logging/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"What a log line should record, how to give it a structure, what must stay out of it, and why logs grow fastest when a system is in trouble.","formats":{"gemtext":"https://engineer.company/glossary/logging/index.gmi","gopher":"https://engineer.company/glossary/logging/gophermap","html":"https://engineer.company/glossary/logging/","json":"https://engineer.company/glossary/logging/index.json","markdown":"https://engineer.company/glossary/logging/index.md","pdf":"https://engineer.company/glossary/logging/index.pdf"},"kind":"page","language":"en","title":"What is logging?","url":"https://engineer.company/glossary/logging/","version":1}