# Site Reliability & Monitoring

> Selected work demonstrating this service.

- [Developed and launched the company's first observability dashboard, providing real‑time system performance insights and data visualization on the large office TV.](https://engineer.company/portfolio/developed-and-launched-the-company-s-first-observability-9/)
- [Designed, deployed, and maintained 10 PostgreSQL and MS SQL servers on Ubuntu Linux VPS, ensuring optimal server performance and reliability.](https://engineer.company/portfolio/designed-deployed-and-maintained-10-postgresql-and-ms-13/)
- [Mitigated operational risks by implementing a monitoring dashboard using Grafana and Prometheus, improving system reliability.](https://engineer.company/portfolio/mitigated-operational-risks-by-implementing-a-monitoring-dashboard-22/)
- [Administered 40 websites on Ubuntu Linux hosting servers with Apache and Nginx, ensuring high availability and performance.](https://engineer.company/portfolio/administered-40-websites-on-ubuntu-linux-hosting-servers-53/)
- [Architected, developed, implemented, supported infrastructure, data processing, and the map application for 2 years non‑stop without any weekends, holidays, or vacations, 10–14 hours a day.](https://engineer.company/portfolio/architected-developed-implemented-supported-infrastructure-data-processing-and-55/)
- [Built email as a platform capability — three providers with failover, delivery webhooks, send and delivery logging, templating and campaigns — behind a startup check that will not boot without one.](https://engineer.company/portfolio/built-email-as-a-platform-capability-with-failover-62/)
- [Owned end‑to‑end deployments of the platform to Azure, managing releases across development, staging and production environments.](https://engineer.company/portfolio/owned-end-to-end-deployments-of-the-platform-71/)
- [Provided round‑the‑clock 24/7 infrastructure support for an IPTV/OTT streaming platform, administering ~1,000 servers plus client‑owned systems for global customers in China, the US and Germany.](https://engineer.company/portfolio/provided-round-the-clock-24-7-infrastructure-support-82/)
- [Ensured uninterrupted delivery of IPTV streaming signals between suppliers and clients, monitoring and maintaining the streaming network and IP telephony around the clock.](https://engineer.company/portfolio/ensured-uninterrupted-delivery-of-iptv-streaming-signals-between-83/)
- [Planned and implemented new infrastructure functionality for internal and external systems, building solutions durable enough to still run years later with minimal change.](https://engineer.company/portfolio/planned-and-implemented-new-infrastructure-functionality-for-internal-85/)
- [Moved slow work off the request path onto a River job queue — 15 worker modules, 8 scheduled tasks and 20 pg_cron jobs — so a request returns while the work behind it carries on.](https://engineer.company/portfolio/moved-slow-work-onto-a-river-job-queue-97/)
- [Built first‑party error monitoring and OpenTelemetry tracing rather than buying them — payload sanitising, spike and regression detection, symbolication and a synthetic heartbeat — behind 11 operator views.](https://engineer.company/portfolio/built-first-party-error-monitoring-and-tracing-98/)
- [Built fail‑closed abuse controls — 22 Redis‑backed rate limiters, Cloudflare Turnstile, request idempotency and an origin lock — so the platform sheds bots and floods instead of trusting its callers.](https://engineer.company/portfolio/built-fail-closed-abuse-controls-and-rate-limiting-100/)
- [Found and closed three SSH brute‑force protections that had never worked: a ban jail watching port 22 while the daemon listened on 1986, a rate limit shadowed by a broader rule above it, and a ban action whose binary never resolved, so no ban had ever applied.](https://engineer.company/portfolio/found-three-ssh-brute-force-protections-that-never-worked-110/)
- [Proved the intrusion‑banning path end to end on every hardening run by banning a reserved test address, reading the resulting kernel rule and unbanning in a guaranteed cleanup block, so a jail that stops working fails a run instead of reporting healthy.](https://engineer.company/portfolio/proved-the-intrusion-banning-path-on-every-converge-112/)
- [Built encrypted off‑host backups on restic with retention pruning, an integrity check and a monthly automated restore drill, then audited the recovery position and wrote down the gaps rather than leaving them to be found during an incident.](https://engineer.company/portfolio/built-encrypted-backups-with-a-monthly-restore-drill-114/)
- [Built dead‑man's‑switch monitoring that pings only while memory and disk are healthy, so a degraded host raises an alert by going silent — and caught six variable names saying "free" where the check correctly measured "available", an order of magnitude apart on a 464 MB box.](https://engineer.company/portfolio/built-dead-mans-switch-monitoring-115/)
- [Made check mode tell the truth across the whole platform after finding six probes deciding on a value the host never gave, because Ansible's command module reports success under --check while skipping the command entirely.](https://engineer.company/portfolio/made-ansible-check-mode-tell-the-truth-116/)
- [Built seven read‑only reporting roles that render a live host to Markdown — facts, access, git, metrics, traffic, security and provider inventory — under a rule that no number is printed the run did not measure.](https://engineer.company/portfolio/built-seven-read-only-host-reporting-roles-120/)
- [Split the four per‑host operational secrets after establishing that two hosts sharing one dead‑man's switch alert less than two switches, not more, and that a shared backup passphrase makes two hosts one repository.](https://engineer.company/portfolio/split-every-operational-secret-per-host-125/)
- [Stopped an application filling memory at 41 MB a second — a recorded 111 GB of compressed pages on a 36 GB machine — by bounding every event stream, subscribing by event type and putting a rate budget on logging, taking 610,996 log lines down to 1,411.](https://engineer.company/portfolio/stopped-an-application-filling-memory-at-41mb-a-second-155/)

<https://engineer.company/services/site-reliability/>
