{"canonical_url":"https://engineer.company/glossary/observability/","content_markdown":"# Observability\n\n## What it is\n\nObservability is the degree to which a team can understand what a running\nsystem is doing from the outside, using only the data the system sends out,\nwithout opening it up or knowing in advance what went wrong. The OpenTelemetry\nproject's primer puts it in terms of questions: an observable system lets you\nask questions about it without knowing its inner workings, including questions\nnobody thought of when the software was written. The primer's shorthand for the\nquestion it should be able to answer is \"Why is this happening?\"\n\nThat makes observability a property of the system, not a product. A tool can\nstore and display data, but the data has to come from the application first.\nOpenTelemetry's documentation is direct about it: for a system to be observable,\nits components must be instrumented, meaning their code emits signals. It then\nsets a practical bar. An application counts as properly instrumented when its\nexisting signals let developers investigate an issue without first changing the\ninstrumentation.\n\nThe data is called telemetry, and OpenTelemetry sorts it into signals:\n\n- **Traces** record the path of one request through the system. A trace is made\n  of spans, each a single unit of work with a name, a start and end time and\n  attributes such as the HTTP method and route. The first span is the root and\n  covers the request from start to finish.\n- **Metrics** are measurements captured at runtime and aggregated over time:\n  error rate, request rate, processor use.\n- **Logs** are timestamped records of events. Many of them are not tied to any\n  particular request.\n- **Baggage** is contextual information passed along with a request so that\n  other signals can use it.\n\nThe signals page also lists events, a specific type of log, and profiles,\nrecordings of resource usage at the level of the code, as under development or\nat the proposal stage.\n\nWhat turns three piles of data into something a team can question is\ncorrelation. OpenTelemetry propagates context between services, so a span in\none service knows which trace it belongs to, and it can stamp the IDs of the\nactive trace and span onto each log record. Every signal also carries a\nresource: attributes that identify the entity producing it, such as the service\nname, the host or the container. With those shared keys, a team can move from a\nmetric to the requests behind it, and from a request to the log lines it\nproduced.\n\nObservability and monitoring overlap but are not the same thing. Monitoring\nwatches for conditions someone decided in advance to care about and raises an\nalert when one appears. Observability is what the team has to work with once the\nalert has fired and the cause is something nobody planned for.\n\n## Why it matters\n\nModern systems fail in ways that are hard to predict. A request may pass through\na gateway, several services, a queue and a database before it answers, and a\nfault in any one of them can show up as slowness somewhere else. The primer\ncalls distributed tracing essential for such systems, because their problems\nare often nondeterministic or too complicated to reproduce on a developer's\nmachine. When a failure cannot be reproduced, the telemetry recorded while it\nhappened is the only evidence there is.\n\nDashboards and alerts answer questions that were anticipated. That is valuable,\nand it covers the failures a team has seen before. The hard incidents are the\nnew ones, where the question is not whether error rates rose but why requests\nfrom one region, through one version, to one payment provider began to fail. If\nanswering that means writing new code, deploying it and waiting for the problem\nto come back, the outage lasts as long as that round trip.\n\nObservability also keeps the user in view. The primer defines reliability as\nwhether the service is doing what users expect, and gives an example: a shop\ncould be up all the time and still be unreliable if clicking to add black shoes\nto the cart does not always add black shoes. A host metric would never show\nthat. Request-level telemetry can, and it is what service level indicators are\nbuilt from: measurements taken from the user's point of view, which service level\nobjectives then tie to what the business needs.\n\nFinally, it changes the cost of an incident. A team that can follow a slow\nrequest to the exact call that held it up spends its time fixing, not guessing,\nand the postmortem afterwards can quote evidence rather than recollection.\n\n## What it looks like in practice\n\nTake an invented online shop. Since 14:00, a small share of checkouts take more\nthan two seconds. Nothing is down, no alert has fired, and no dashboard was\nbuilt for this. Here is how a team with correlated signals would work through\nit. Every name, ID and number below is made up for the example.\n\nThe first step is the latency histogram for checkout, measured in seconds. In\nthe OpenMetrics text format, a bucket can carry an exemplar after a `#`: a\nreference to data outside the metric, most commonly the ID of a trace that\nlanded in that bucket. Two of the buckets read:\n\n```text\ncheckout_bucket{le=\"1.0\"} 95\ncheckout_bucket{le=\"2.5\"} 97 # {trace_id=\"3f1c9a77e2b04d18a6c5e0b9d4127f60\"} 2.3\n```\n\nThe second step follows that trace ID. The trace shows where the 2.3 seconds\nwent, span by span:\n\n```text\nPOST /checkout                       2.30 s   service checkout\n  SELECT cart                        0.04 s   service checkout\n  POST /authorise                    2.21 s   service payments\n    event: provider timeout, retrying\n  INSERT order                       0.03 s   service checkout\n```\n\nThe third step pulls the log lines that carry the same trace ID. OpenTelemetry's\nspecification for logs written in formats other than its own protocol asks that\nthe IDs be recorded as top-level fields named `trace_id` and `span_id` in JSON,\nlowercase and hex-encoded:\n\n```json\n{\n    \"timestamp\": 1793110051.402,\n    \"severity\": \"WARN\",\n    \"body\": \"provider timeout after 2000 ms, retrying\",\n    \"provider\": \"acquirer-b\",\n    \"trace_id\": \"3f1c9a77e2b04d18a6c5e0b9d4127f60\",\n    \"span_id\": \"a91c04e7d2f3b658\"\n}\n```\n\nThree hops, and the question nobody planned for has an answer: one of two\npayment providers began timing out, and every checkout routed to it waited for\na retry. The same attribute, `provider`, can now be grouped on across all\nrecent traces to confirm the pattern before anyone touches the code.\n\nNone of that works if the services do not identify themselves. OpenTelemetry\nSDKs set the `service.name` resource attribute to `unknown_service` by default,\nand the documentation recommends setting it explicitly, in code or through an\nenvironment variable. Other resource attributes can be added the same way:\n\n```sh\nenv OTEL_SERVICE_NAME=checkout \\\n    OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production \\\n    ./checkout\n```\n\nOn this site, three portfolio entries show observability being built up in\nsteps. One describes\n[a company's first observability dashboard](/portfolio/built-the-companys-first-observability-dashboard/),\nwhich collected system metrics such as CPU, memory and disk from remote Linux\nservers over SSH; a second version was later built on Grafana and Prometheus.\nWork to\n[cut operational risk with Grafana and Prometheus](/portfolio/cut-operational-risk-with-grafana-and-prometheus/)\ninstrumented the key services and chose metrics that meant something over vanity\nnumbers. A later entry,\n[first-party error monitoring and tracing](/portfolio/built-first-party-error-monitoring-and-tracing/),\nadded OpenTelemetry tracing over OTLP, so that a slow request can be followed\nacross the frontend, the API and the database rather than guessed at, alongside\nan error pipeline that sanitises payloads before they are stored.\n\n## Where it goes wrong\n\n**Buying it instead of building it in.** A new tool with nothing useful flowing\ninto it is an empty room. Observability comes from instrumentation in the code\nand the libraries, so budget the engineering time to add spans, attributes and\nlog fields, not only the licence.\n\n**Three signals, three islands.** Metrics in one tool, logs in another and\ntraces in a third, with no shared key between them, leave the team copying\ntimestamps from one screen to another. Put the trace and span IDs into every log\nrecord, attach exemplars to latency histograms where the stack supports them,\nand give every signal the same resource attributes.\n\n**Every service called unknown_service.** It is the SDK default, and a fleet of\nservices sharing it cannot be told apart. Set `OTEL_SERVICE_NAME` for every\nservice. The resource is attached when the tracer and metric providers are\ncreated and cannot be changed afterwards, so it has to be right at start-up.\n\n**Only the edges are instrumented.** Zero-code instrumentation is a good start\nand records what happens at the edges of an application: in its libraries and\nthe environment it runs in. The questions in an incident are usually about the\nbusiness: which customer tier, which provider, which feature flag value. Those\nneed attributes added in code. OpenTelemetry's documentation calls code-based\ninstrumentation a complement to zero-code, and the two can run at once.\n\n**Instrumenting during the incident.** If the answer required a new log line and\na deploy, the system was not properly instrumented by the primer's own measure.\nMake \"what did we have to add?\" a standing question in every incident review,\nand treat each answer as a follow-up item.\n\n**Unbounded detail in the wrong signal.** User IDs, full URLs and request bodies\nbelong on spans and log records, not in metric labels, where every new value\ncreates another time series. The OpenMetrics standard caps the label names and\nvalues on an exemplar at 128 characters in total, precisely to stop exemplars\nbeing used to carry span data, and lets an ingestor discard exemplars\naltogether. Check that the links survive the whole path from the exporter to the\nscreen before relying on them.\n\n**Telemetry that leaks.** Spans and logs record what requests contained, and\nthat can include addresses, names and tokens. Decide which attributes may be\nrecorded, strip the rest before storage, and protect telemetry with the same\ncare as the database it describes.\n\n## Further reading\n\n- [Observability primer](https://opentelemetry.io/docs/concepts/observability-primer/)\n  — the OpenTelemetry project's definition, with reliability, SLIs and SLOs,\n  spans and distributed traces.\n- [Signals](https://opentelemetry.io/docs/concepts/signals/) — traces, metrics,\n  logs and baggage, and the signals still under development.\n- [Instrumentation](https://opentelemetry.io/docs/concepts/instrumentation/) —\n  code-based and zero-code instrumentation, and how context propagation ties\n  signals together.\n- [Resources](https://opentelemetry.io/docs/concepts/resources/) — resource\n  attributes, the `unknown_service` default and the environment variables that\n  set them.\n- [Trace Context in non-OTLP Log Formats](https://opentelemetry.io/docs/specs/otel/compatibility/logging_trace_context/)\n  — the stable specification for recording trace and span IDs in syslog, plain\n  text and JSON logs.\n- [OpenMetrics 1.0](https://prometheus.io/docs/specs/om/open_metrics_spec/) —\n  the exposition format standard, including exemplars and its advice on\n  cardinality.\n\n\u003chttps://engineer.company/glossary/observability/\u003e\n","date_modified":"2026-10-30T02:51:31Z","description":"What observability is, how traces, metrics and logs are tied together by shared IDs, and the habits that leave a system hard to question.","formats":{"gemtext":"https://engineer.company/glossary/observability/index.gmi","gopher":"https://engineer.company/glossary/observability/gophermap","html":"https://engineer.company/glossary/observability/","json":"https://engineer.company/glossary/observability/index.json","markdown":"https://engineer.company/glossary/observability/index.md","pdf":"https://engineer.company/glossary/observability/index.pdf"},"kind":"page","language":"en","title":"What is observability?","url":"https://engineer.company/glossary/observability/","version":1}