# Built first‑party error monitoring and OpenTelemetry tracing rather than buying them — payload sanitising, spike and regression detection, symbolication and a synthetic heartbeat — behind 11 operator views.

2025

**Situation.** The difference between a platform that is up and a platform that is working is whether anyone would know. An error a user hits at eleven at night, on a page nobody tests, is invisible unless something goes and collects it. The usual answer is to buy a hosted error tracker, which is a good answer — and it also means the platform's own errors, stack traces and user context leave for a third party.

**Task.** Errors and traces had to be collected, grouped and made actionable, without the platform's internals leaving the platform.

**Action.** Two pieces got built. OpenTelemetry handles tracing over OTLP, so a slow request can be followed across the frontend, the API and the database rather than guessed at. Alongside it sits an in‑house error pipeline — an errmon service and an ingest service — that sanitises payloads before storage, groups errors into recurring problems rather than a flat list, detects spikes and regressions with a cooldown so one bad deploy does not page anyone forty times, symbolicates minified frontend stack traces back into readable code, and runs a synthetic heartbeat to prove that the pipeline itself is alive. It all lands in the database as error events, error groups, an inbox, API latency and stack samples, and it surfaces through 11 operator views, including one for service level objectives.

**Result.** Errors turn into a queue that someone can work through, and a regression announces itself instead of being discovered by a user. Building rather than buying cost real time and means this is one more thing to maintain — a bought tracker would have been running the same afternoon. What it bought was that nothing sensitive leaves, and that the alerting rules fit this platform rather than a generic one.

---

- Role: Site Reliability Engineer
- Categories: [Backend Engineering](https://engineer.company/categories/backend/), [DevOps](https://engineer.company/categories/devops/), [Infrastructure](https://engineer.company/categories/infrastructure/), [Reliability & Backups](https://engineer.company/categories/reliability/), [Monitoring & Observability](https://engineer.company/categories/observability/), [Security](https://engineer.company/categories/security/)
- Services: [Backend & API Development](https://engineer.company/services/backend-development/), [DevOps & CI/CD Automation](https://engineer.company/services/devops-cicd/), [Site Reliability & Monitoring](https://engineer.company/services/site-reliability/)

<https://engineer.company/portfolio/built-first-party-error-monitoring-and-tracing-98/>
