Knowing before your customers do

Selected work demonstrating this service.

• Built the company's first observability dashboard

  Developed and launched the company's first observability dashboard, providing real‑time system performance insights and data visualization on the large office TV.

• Ran 10 PostgreSQL and MS SQL servers on Ubuntu

  Designed, deployed, and maintained 10 PostgreSQL and MS SQL servers on Ubuntu Linux VPS, ensuring optimal server performance and reliability.

• Cut operational risk with Grafana and Prometheus

  Mitigated operational risks by implementing a monitoring dashboard using Grafana and Prometheus, improving system reliability.

• Administered 40 websites on Apache and Nginx

  Administered 40 websites on Ubuntu Linux hosting servers with Apache and Nginx, ensuring high availability and performance.

• A mapping product, carried end to end

  Architected, developed, implemented, supported infrastructure, data processing, and the map application for 2 years non‑stop without any weekends, holidays, or vacations, 10–14 hours a day.

• Built email with failover and delivery logging

  Built email as a platform capability — three providers with failover, delivery webhooks, send and delivery logging, templating and campaigns — behind a startup check that will not boot without one.

• Owned end‑to‑end platform deployments to Azure

  Owned end‑to‑end deployments of the platform to Azure, managing releases across development, staging and production environments.

• Ran 24/7 support for ~1,000 IPTV/OTT servers

  Provided round‑the‑clock 24/7 infrastructure support for an IPTV/OTT streaming platform, administering ~1,000 servers plus client‑owned systems for global customers in China, the US and Germany.

• Kept IPTV streaming signals running non‑stop

  Ensured uninterrupted delivery of IPTV streaming signals between suppliers and clients, monitoring and maintaining the streaming network and IP telephony around the clock.

• Built infrastructure that still runs years later

  Planned and implemented new infrastructure functionality for internal and external systems, building solutions durable enough to still run years later with minimal change.

• Moved slow work onto a River job queue

  Moved slow work off the request path onto a River job queue — 15 worker modules, 8 scheduled tasks and 20 pg_cron jobs — so a request returns while the work behind it carries on.

• Built first‑party error monitoring and tracing

  Built first‑party error monitoring and OpenTelemetry tracing rather than buying them — payload sanitising, spike and regression detection, symbolication and a synthetic heartbeat — behind 11 operator views.

• Built fail‑closed rate limiting and anti‑abuse

  Built fail‑closed abuse controls — 22 Redis‑backed rate limiters, Cloudflare Turnstile, request idempotency and an origin lock — so the platform sheds bots and floods instead of trusting its callers.

• Three locks on the door, none of them locked

  Found and closed three SSH brute‑force protections that had never worked: a ban jail watching port 22 while the daemon listened on 1986, a rate limit shadowed by a broader rule above it, and a ban action whose binary never resolved, so no ban had ever applied.

• Proved the intrusion ban works every run

  Proved the intrusion‑banning path end to end on every hardening run by banning a reserved test address, reading the resulting kernel rule and unbanning in a guaranteed cleanup block, so a jail that stops working fails a run instead of reporting healthy.

• Backups that have actually been restored

  Built encrypted off‑host backups on restic with retention pruning, an integrity check and a monthly automated restore drill, then audited the recovery position and wrote down the gaps rather than leaving them to be found during an incident.

• Built dead‑man's‑switch host monitoring

  Built dead‑man's‑switch monitoring that pings only while memory and disk are healthy, so a degraded host raises an alert by going silent — and caught six variable names saying "free" where the check correctly measured "available", an order of magnitude apart on a 464 MB box.

• Made Ansible check mode tell the truth

  Made check mode tell the truth across the whole platform after finding six probes deciding on a value the host never gave, because Ansible's command module reports success under --check while skipping the command entirely.

• Built 7 read‑only host reporting roles

  Built seven read‑only reporting roles that render a live host to Markdown — facts, access, git, metrics, traffic, security and provider inventory — under a rule that no number is printed the run did not measure.

• Split every operational secret per host

  Split the four per‑host operational secrets after establishing that two hosts sharing one dead‑man's switch alert less than two switches, not more, and that a shared backup passphrase makes two hosts one repository.

• An app that ate 41 MB a second, stopped

  Stopped an application filling memory at 41 MB a second — a recorded 111 GB of compressed pages on a 36 GB machine — by bounding every event stream, subscribing by event type and putting a rate budget on logging, taking 610,996 log lines down to 1,411.

• Measured 13 addresses with no tracking

  Measured thirteen addresses from the web server's own access log alone — no cookie, no tracker and no third‑party service — masking the client network as each line is written, and folding the result into a permanent archive of 21 daily counts and 20 monthly facets.

• Made six collectors state what they skipped

  Made six reporting collectors state the part of their input they never read — files that stayed shut, probes not installed, journal lines unrecognised — and fitted thresholds to 116 measured samples, after establishing that a blind collection and a healthy host render the same page.