{"canonical_url":"https://engineer.company/glossary/latency/","content_markdown":"# Latency\n\n## What it is\n\nLatency is how long a system takes to start responding: the time from a request\nleaving the client to the first byte of the reply coming back. Engineers also\ncall it delay, lag or, for a network path measured there and back, round-trip\ntime (RTT); in web performance the same idea goes by time to first byte (TTFB).\nResponse time is its larger sibling, running to the last byte of the answer.\n\nIt is easy to confuse with throughput and bandwidth, and the three answer\ndifferent questions. Bandwidth is how many bits a link can carry per second.\nThroughput is how much work a system completes per second. Latency is how long\none piece of work waits. A nightly batch job can have excellent throughput and\nterrible latency, and a chat server the reverse.\n\nEvery millisecond of it comes from one of a few places:\n\n- **propagation**, the signal crossing the distance: light in optical fibre\n  covers roughly 200 km per millisecond, so a 1,000 km path costs at least 10 ms\n  there and back before any software runs;\n- **transmission**, putting the bytes on the wire, which matters for large\n  payloads on slow links;\n- **queueing**, waiting behind other work in a router buffer, a thread pool, a\n  connection pool or a lock, the one component that grows with load;\n- **processing**, the actual work: the query, the template, the encryption;\n- **protocol round trips**, the conversation before the first useful byte. A new\n  HTTPS connection over TCP needs a TCP handshake and a TLS handshake first, and TLS 1.3\n  cut its full handshake to a single round trip.\n\nThe IETF's IP Performance Metrics group pinned the network version down in RFC\n2681: round-trip delay runs from the first bit of a packet sent to the last bit\nof the reply received. Two of its choices carry over to any service. A reply\nthat never arrives is counted as undefined, informally infinite, rather than\nquietly left out of the sample. And the statistics it describes are\npercentiles, the median and the minimum. It also observes that the minimum\nreflects propagation and transmission alone, while everything above the minimum\nindicates congestion, a split that holds for an API as well as for a path\nbetween routers.\n\nFor a single web request, curl will break the wait into its stages:\n\n```sh\ncurl -s -o /dev/null \"$URL\" -w '\ndns        %{time_namelookup}\nconnect    %{time_connect}\ntls        %{time_appconnect}\nfirst byte %{time_starttransfer}\ntotal      %{time_total}\n'\n```\n\nEach figure is seconds since the start, so the gaps between them are the costs:\nlooking up the name, opening the connection, the TLS handshake, and then the\nserver's own thinking time up to the first byte.\n\n## Why it matters\n\nLatency is what a user feels; throughput is what the server bill reflects. When\na page is slow, the decision is where to spend. The five components above need\ndifferent fixes; four of them show why buying hardware is not a general remedy.\nDistance is fixed by moving closer: a cache, a CDN, a region near the users.\nRound trips are fixed by having fewer of them: reusing connections, batching\nqueries, fetching in parallel. Processing is fixed by doing less work: an index,\na better query plan. Adding capacity can reduce queueing; faster processors, more memory or better I/O\nmay also reduce processing time when those resources are the bottleneck. A larger\nserver is not automatically a faster one: measure where the time goes before\nchoosing hardware.\n\nAverages make the problem look smaller than it is, because users meet the tail.\nA page that calls ten back-end services in parallel waits for the slowest. If\neach call independently has a 1% chance of being slow, then 1 − 0.99¹⁰, about 9.6% of page loads,\nhit at least one slow call. A rare delay in the parts becomes a common one in\nthe whole.\n\nThat is why serious targets are written in percentiles, for example 99% of\nrequests answered within 300 ms over 30 days, and why the 99th percentile is\nusually the number worth watching. The median says how the service feels on a\ngood day; the high percentiles say how it feels to the customers who are about\nto leave.\n\n## What it looks like in practice\n\nOn a geographical map application, latency in rendering map layers and querying\nlocation data was hurting both users and the reliability of the backend. The\nwork focused on spatial query processing in a backend that used a legacy\nMicrosoft SQL Server database.\n[Moving to PostgreSQL with PostGIS cut average response time from 2.5 seconds to under 250 milliseconds](/portfolio/improved-gis-map-performance-10x-on-postgresql/),\na 10x improvement in spatial query performance. The work behind it was GiST and\nSP-GiST indexes on the spatial columns, more than 50 million records migrated,\nand configuration tuned for concurrent access; backend CPU load dropped by 65%.\n\nLatency also caps throughput where one row is contended. A note on\n[selling the last ticket exactly once](/notes/selling-limited-stock-without-overselling/)\nputs the ceiling for a single item at roughly the sales one lock carries divided\nby how long the lock is held. Anything slow inside the lock lowers that ceiling:\na payment call made while holding it brings it down to about two sales a\nsecond. Even a safety measure has a price here. A synchronous standby in\nPostgreSQL adds latency to every commit, and that lengthens exactly the lock\nhold time the ceiling divides by.\n\nAnd to find where the time goes, the\n[first-party monitoring on one platform](/portfolio/built-first-party-error-monitoring-and-tracing/)\nrecords API latency in its own database and traces requests over OpenTelemetry,\nso a slow request can be followed across the frontend, the API and the database\nrather than guessed at.\n\n## Where it goes wrong\n\n**The average hides the tail.** A mean of 100 ms is compatible with one request\nin fifty taking three seconds. Record latency as a histogram and read its\npercentiles. Then mind how they combine: the 99th percentiles of five servers\ncannot be averaged into the fleet's 99th percentile. Prometheus makes the point\nin its own documentation, where quantiles computed by a summary cannot be\naggregated, while histogram buckets can be summed first and the quantile taken\nafterwards.\n\n**The timer starts in the wrong place.** A server that measures from the moment\nits handler begins misses the time a request spent in the load balancer, the\naccept queue or the wait for a pooled connection. Load generators that wait for\neach answer before sending the next request send fewer requests exactly when\nthe system stalls, so the stall barely shows in the results. Measure from the\nclient as well, and let the server report its share in a standard header:\n\n```http\nServer-Timing: db;dur=53, app;dur=47.2\n```\n\n**The conversation is too chatty.** A page that runs one query per row of a\nlist pays one round trip per row. With the database in another region at 30 ms\na round trip, a hundred rows cost three seconds before any query does real\nwork. One query with a join, or one batched call, pays the distance once.\n\n**Retries multiply the load.** A client that gives up after one second and\nretries three times turns one slow server into four times the traffic, which\nmakes it slower still. Give each request an overall deadline, back off\nexponentially with random jitter between attempts, and cap the number of\nretries.\n\n## Further reading\n\n- [RFC 2681: A Round-trip Delay Metric for IPPM](https://www.rfc-editor.org/rfc/rfc2681)\n  — round-trip delay defined to the bit, with lost replies and percentiles\n  handled explicitly.\n- [RFC 7679: A One-Way Delay Metric for IP Performance Metrics](https://www.rfc-editor.org/rfc/rfc7679)\n  — the one-way counterpart, published as an Internet Standard.\n- [Server Timing](https://www.w3.org/TR/server-timing/) — the W3C header that\n  lets a server tell the browser where its time went.\n- [Histograms and summaries](https://prometheus.io/docs/practices/histograms/)\n  — why quantiles from a summary cannot be combined and histogram buckets can.\n- [RFC 8446: TLS 1.3](https://www.rfc-editor.org/rfc/rfc8446) — the handshake\n  design, including the round trip it removes.\n\n\u003chttps://engineer.company/glossary/latency/\u003e\n","date_modified":"2026-11-12T05:34:23Z","description":"What latency measures, why percentiles beat averages, how round trips and queues add up, and how to find a slow request before buying faster hardware.","formats":{"gemtext":"https://engineer.company/glossary/latency/index.gmi","gopher":"https://engineer.company/glossary/latency/gophermap","html":"https://engineer.company/glossary/latency/","json":"https://engineer.company/glossary/latency/index.json","markdown":"https://engineer.company/glossary/latency/index.md","pdf":"https://engineer.company/glossary/latency/index.pdf"},"kind":"page","language":"en","title":"What is latency?","url":"https://engineer.company/glossary/latency/","version":1}