OpenTelemetry

What it is

OpenTelemetry is an open-source standard, with libraries and tools to match, for producing traces, metrics and logs from running software and handing them to whatever system stores and displays them. Most engineers call it OTel. It is a project of the Cloud Native Computing Foundation, formed when two earlier efforts, OpenTracing and OpenCensus, merged into one.

The most useful thing to know about it is what it leaves out. OpenTelemetry is not an observability backend: it keeps no data, draws no dashboards and sends no alerts. Jaeger, Prometheus and the commercial platforms do that part. So the question “OpenTelemetry or Prometheus?” compares a pipe with a destination, and the usual answer is both: OpenTelemetry where the data is produced, a backend of the team’s choosing where it is kept.

The project is several pieces that fit together:

• a specification that defines the API, the SDK and the data, across every language;
• OTLP, the protocol that carries telemetry over gRPC or HTTP, encoded with Protocol Buffers;
• semantic conventions, shared names for shared things, so an HTTP route or a database statement is labelled the same way in Go, Java and Python;
• an API and SDK per language, with instrumentation libraries and automatic instrumentation for popular frameworks;
• the Collector, a separate process that receives telemetry, processes it and exports it onward.

Of the signals it carries, traces take the most explaining. A trace follows one request through every service it touches. Each step is a span: a name, a start time, a duration, a set of attributes, and the identity of the span that caused it. When the request crosses from one service to the next, that identity travels with it in a header, by default the one W3C Trace Context defines:

traceparent: 00-a0892f3577b34da6a3ce929d0e0e4736-f03067aa0ba902b7-01

The four fields are the format version, the trace ID shared by every span of the request, the ID of the calling span, and a set of flags, where a final 01 marks the trace as sampled. Metrics and logs complete the picture, and a fourth signal, baggage, carries small key-value facts along with the request so that a service three hops away can still see them.

Why it matters

Storage is the visible cost of observability. The stickier one is instrumentation: someone has to add the calls to every service, in every language, and keep them current as the code changes. When each vendor shipped its own agent and its own API, changing vendor meant doing all of that again. With OpenTelemetry the code calls one API, and the choice of backend moves into configuration, either an exporter in the SDK or a route in the Collector. The project states the principle outright: the data a system generates belongs to whoever runs it.

That ownership is practical as well as contractual. The Collector sits between the applications and the backend, which makes it the place to filter, sample, rename or strip attributes before anything leaves, and to send the same stream to two backends while a migration is under way. Moving from one vendor to another becomes a change to a Collector configuration rather than to a hundred services.

Volume is the cost to plan for. Each request can produce dozens of spans, and at real traffic the bill for keeping all of them can outgrow the service being watched. The SDK’s default sampler, ParentBased(root=AlwaysOn), samples every new trace and follows the caller’s decision for the rest, so out of the box everything is kept. How much to keep is a decision a team makes on purpose.

Maturity differs by signal and by language. OTLP is stable for traces, metrics and logs, while each language’s SDK moves at its own pace, and the project’s status page says where each one stands. Read it before committing a service written in a less common language.

And it is not always worth it. A single application with structured logs and a handful of metrics gains little from distributed tracing. OpenTelemetry pays for itself when a request crosses process boundaries, through a frontend, an API, a queue and a database, and the question “where did the time go?” takes a meeting to answer.

What it looks like in practice

On one platform the monitoring was built rather than bought, and OpenTelemetry is its tracing half. Tracing runs over OTLP, so a slow request is followed across the frontend, the API and the database rather than guessed at. Beside it runs an in-house error pipeline, an errmon service and an ingest service, that sanitises payloads before they are stored, groups errors into recurring problems, detects spikes and regressions with a cooldown so one bad deploy does not page anyone forty times, symbolicates minified frontend stack traces, and runs a synthetic heartbeat to prove that the pipeline itself is alive.

The results land in the platform’s own database as error events, error groups, an inbox, API latency and stack samples. Operators work from 11 views over that data, one of them for service level objectives. The page is plain about the price: building cost real time and added one more thing to maintain, and a bought tracker would have been running the same afternoon. In return, sensitive data stays on the platform. The alerting rules, too, were written for this platform rather than a generic one.

Where it goes wrong

The trace stops at a boundary. Context crosses HTTP calls easily, because the instrumentation libraries write and read the header on both sides. A message queue, a scheduled job or a proxy that strips unknown headers breaks the chain unless someone carries the context across by hand. The trace splits in two, and the slow half looks unrelated to the request that caused it. Put the context in the message metadata, and check that every service runs the same propagator.

Attributes leak or explode. A span attribute is easy to add, and the easiest values to add are the wrong ones: email addresses, full URLs with tokens in the query string, SQL with the literal values still in it. On metrics the danger is cardinality. An attribute with millions of distinct values, a user ID for instance, multiplies the number of time series and the bill along with it. Follow the semantic conventions, keep identifiers off metrics, and redact in the Collector as a second line of defence.

Sampling happens by accident. Head sampling decides at the start of a request, before anyone knows whether it will be slow or fail, so keeping 1% of traces also keeps about 1% of the rare slow ones. Tail sampling decides after seeing a trace’s spans together and can keep every error and every slow request, but the component doing it must hold all that data while it waits. It is a stateful service with capacity of its own to plan and to monitor.

The pipeline is trusted blindly. OTLP acknowledges delivery one hop at a time; end-to-end delivery is outside its scope. A Collector that runs short of memory or falls behind on its export queue drops telemetry, and the applications sending to it carry on as if all were well. Watch the Collector’s own metrics, run more than one, and treat a gap in the traces as a fault to explain rather than a quiet hour.

Further reading

• What is OpenTelemetry? — the project’s own definition, its components, and what it deliberately is not.
• OTLP Specification — the wire protocol: transports, encoding, compression and acknowledgement.
• Trace Context — the W3C Recommendation behind the traceparent header.
• Context propagation — how a trace follows a request across process boundaries.
• Semantic conventions — the shared names for attributes, metrics and resources.
• Sampling — head and tail sampling, and what each one costs to run.