{"canonical_url":"https://engineer.company/notes/selling-limited-stock-without-overselling/","content_markdown":"# Selling the last ticket exactly once\n\n2026-10-13\n\nA flash sale puts ten thousand units in front of a million people at once. The\ninteresting failure is not that the shop goes down. It is that the shop stays\nup and sells ten thousand and forty. The instinct that follows — the database\nis too small, shard it — answers a question nobody asked. The database was\nnever too big. Every buyer was writing to\nthe same row *(the whole incident lives in one row)*.\n\nWe take the problem apart here before it reaches a project: where the ceiling\ncomes from, the one statement that removes most of it, and what to build when a\nsingle database really is not enough.\n\n## One row, every buyer\n\nTwenty million rows spread over a cluster is a solved problem. One row that\nevery request has to change is not, and adding machines does not touch it,\nbecause the row lives on one of them. Distributed stores say so plainly:\nDynamoDB gives a single partition about 1,000 write units a second, and every\nwrite to one item lands on that item's partition however many others there\nare.\n\nThere is a theorem behind the shape. Bailis and colleagues call an invariant\ninvariant-confluent when replicas can accept writes on their own and still merge\ninto a valid state. `stock \u003e= 0` is the textbook case of one that is not: two\nreplicas that each see one unit left will each sell it, and no merge can take\nthe second sale back. Never overselling needs coordination. The only question\nis how much of it each sale pays for.\n\n## The number that decides it\n\nThe ceiling for one item is roughly how many sales one lock carries, divided by\nhow long the lock is held:\n\n```text\nmax sales/s per item ≈ sales per lock ÷ lock hold time\n```\n\n| What holds the row        | Ceiling per row |\n| ------------------------- | --------------- |\n| Lock, then call payment   | ~2/s            |\n| Lock, then app logic      | ~100–200/s      |\n| One conditional `UPDATE`  | ~500–2,000/s    |\n| Batches of 100 per commit | ~50,000/s       |\n| One in-memory writer      | ~100,000+/s     |\n\nThese are orders of magnitude, not benchmarks; the hardware moves them. What it\ndoes not move is the shape. Most of a short lock is the commit waiting for its\nlog to reach the disk, which is why batching wins by two orders: a hundred sales\nshare one flush.\n\nThat leaves four levers, and every design below pulls one of them:\n\n1. **Hold the lock for less time** — one statement, nothing on the network\n   inside it.\n2. **Do more per lock** — batch the decrements.\n3. **Have more locks** — split the stock across several keys.\n4. **Send fewer requests to the lock** — turn most buyers away before they\n   reach it.\n\nThe last one matters more than it sounds. With a million buyers and ten\nthousand units, 990,000 requests have to fail. Rejections are most of the\ntraffic, so \"sold out\" has to be the cheapest answer the system gives: a waiting\nroom in front of the sale, a gate that admits about as many buyers as there are\nunits, a `sold_out` flag the CDN serves without asking anyone, and a rate limit\nper account.\n\n## Fix the one database first\n\nThe version that crawls, or oversells, usually looks like this:\n\n```sql\nBEGIN;\nSELECT available FROM inventory\n WHERE item_id = :item FOR UPDATE;\n-- the application checks the number,\n-- and perhaps calls a payment provider\nUPDATE inventory SET available = :new\n WHERE item_id = :item;\nCOMMIT;\n```\n\nThe lock is right. What it is held across is wrong: every millisecond the\napplication spends between the `SELECT` and the `UPDATE` is a millisecond the\nnext buyer waits. Without the `FOR UPDATE` it is worse. Two buyers read 1, both\nwrite 0, and one unit has been sold twice.\n\nThe fix makes the check and the decrement one statement:\n\n```sql\nUPDATE inventory\n   SET available = available - :qty\n WHERE tenant_id = :tenant\n   AND item_id = :item\n   AND available \u003e= :qty\nRETURNING available;\n-- one row back: reserved. none: sold out.\n```\n\nIn PostgreSQL at READ COMMITTED, a second buyer who reaches the row while the\nfirst holds it waits, then evaluates the `WHERE` clause again against the row\nthe first one committed. If that sale took the last unit, the condition fails\nand the second buyer gets no row back — sold out — rather than a negative\nnumber. At REPEATABLE READ or SERIALIZABLE the second buyer gets a\nserialization failure instead, which is safe but turns contention into retries;\nREAD COMMITTED is the level this statement is written for. MySQL's InnoDB\nreaches the same answer, because an `UPDATE` there reads the latest committed\nrow at any level. Add the constraint as well, so a bug somewhere else fails\nloudly instead of selling stock that does not exist:\n\n```sql\nALTER TABLE inventory\n  ADD CONSTRAINT available_not_negative\n  CHECK (available \u003e= 0);\n```\n\nThen the rest of the single-database work, roughly in order of payoff:\n\n- **Nothing external inside the transaction.** No payment call, no HTTP request,\n  no message published. Those go before the transaction or after it.\n- **Touch the hot row last.** Insert the order and everything else first, so the\n  lock on the contended row lives only for the final statement and the commit.\n- **No optimistic locking on a hot counter.** A version column turns contention\n  into retries, and under a sale the retries are the load.\n- **Serve the reads from somewhere else.** The product page's \"12 left\" can come\n  from a replica or a cache a second behind. Only the sale needs the primary.\n- **Give inventory a database of its own,** so a sale never waits behind the\n  catalogue for the same disk.\n- **Pool the connections.** A hundred thousand buyers should be a few dozen\n  connections through PgBouncer, not thousands of backends queueing for one\n  lock.\n\nSeats are a different shape, because every unit is distinct, and they want a\ndifferent statement. Each seat is a row, and a buyer skips the rows another\ntransaction holds instead of queueing behind them:\n\n```sql\nUPDATE seats\n   SET held_by = :order,\n       held_until = now() + interval '10 min'\n WHERE id = (\n   SELECT id FROM seats\n    WHERE event_id = :event\n      AND sold_at IS NULL\n      AND (held_until IS NULL\n           OR now() \u003e held_until)\n    LIMIT 1\n    FOR UPDATE SKIP LOCKED)\nRETURNING id;\n```\n\nAn expired hold is simply available again, so no sweeper is needed for the\nanswer to be right. And `now()` is the database's clock, so application servers\nwhose clocks disagree cannot disagree about when a hold ran out.\n\nFor many shops this section is the whole fix. Do it before anything below,\nbecause sharding is a one-way door: easy to walk through, expensive to walk\nback.\n\n## Claim first, pay second\n\nA buyer who takes the last unit and then abandons the card form has not bought\nit. So a sale is not one write but a short lifecycle:\n\n```text\nAVAILABLE → HELD, for minutes → SOLD\n                              → RELEASED\n```\n\n- **The hold is keyed by the request, not the attempt.** Every checkout carries\n  an idempotency key, and the holds table carries\n  `UNIQUE (tenant_id, idempotency_key)`, so a retried click gets back the hold\n  it already has instead of a second unit. One PostgreSQL detail bites here:\n  `INSERT ... ON CONFLICT DO NOTHING RETURNING id` returns nothing when the row\n  already existed, so \"no row back\" means \"look it up\", not \"failed\".\n- **Every transition is conditional.** Confirming a sale moves a hold from\n  `held` to `sold` only where it is still `held`, and a release credits the\n  stock back only if its own update changed exactly one row. A release that\n  runs twice — the timeout and the cancel button, say — gives the unit back\n  once.\n- **Holds expire in minutes.** Five to ten is usual. Extend the hold once when\n  payment starts, so a slow card does not lose the unit it is paying for.\n- **Cap the units per buyer at the hold,** so one account cannot hold the whole\n  stock while it decides.\n\nUnderneath all of it sits one line, the most useful in the design:\n\n```text\navailable + held + sold = total\n```\n\nIt holds after every transaction, one query checks it, and when it stops\nholding, something has oversold or leaked a unit — before a customer says so.\n\n## When one database is not enough\n\nSometimes it genuinely is not: many tenants, many sales at once, several\nregions. The structure that scales keeps one rule — every item has exactly one\nowner of its stock, an inventory service, and nothing else decides — and shards\nthat service by `(tenant_id, item_id)`, so one item's writes never span two\nshards. The largest tenants get cells of their own, so their sale cannot starve\nanyone else's. Only the reservation has to be synchronous; everything after it\ncan be a message.\n\nFor the hot item inside a shard there are four tools, and each pulls one of the\nlevers above.\n\n**A gate in front.** A Redis counter, decremented by a Lua script so the check\nand the decrement are one step, admits about as many buyers as there are units\nand turns everyone else away in microseconds. It is not the arbiter. Redis\nreplicates asynchronously, so after a failover the new primary may have\nforgotten some admissions and let a few too many through. The database's\nconditional `UPDATE` still decides, and a reconciler resets the gate from the\ndatabase's number.\n\n**Split the stock.** Ten thousand units can be a hundred buckets of a hundred,\neach its own row or its own shard, with each buyer sent to one: a hundred locks\ninstead of one. The idea is old — O'Neil's escrow transactions in 1986, then\nBarbará and Garcia-Molina's demarcation protocol, then the bounded counters of\nBalegas and colleagues — and its one real cost is the tail. A buyer's bucket\nruns dry while others still have stock, so either try another bucket or move\nthe leftovers, and near the end of a sale collapse back to one. The same idea\ngives each region a quota of its own, so most sales never cross an ocean to\nask.\n\n**One writer per item.** Put every request for an item on one Kafka partition,\nread by one consumer that keeps the count in memory and commits its decisions in\nbatches. One thread and no locks: the LMAX shape. Requests carry an id, and the\nwriter drops one it has already decided, because a queue delivers twice.\n\n**Units as rows.** When units are few and distinct — seats, numbered editions —\ncreate one row each in advance and claim them with `SKIP LOCKED` as above, or\nkeep the unit ids in a Redis set and `SPOP` one. There is no counter left to\nfight over.\n\n## The rest of checkout\n\nPayment, the order and the receipt are separate systems, and no transaction\nspans them. Two-phase commit is the textbook answer and the wrong one here: it\nblocks while any participant is slow, it adds round trips to the hottest path,\nand payment providers do not take part in it. A saga does the job, a sequence\nof local steps, each with a step that undoes it:\n\n```text\n1. reserve stock       undo: release the hold\n2. pending order       undo: cancel the order\n3. extend the hold     (expires on its own)\n4. authorise payment   undo: void it\n5. confirm HELD → SOLD\n6. capture payment\n```\n\nAuthorising first and capturing last means a payment that arrives after its\nhold has expired is voided rather than refunded: the money never moved. Each\nstep writes its message to an outbox table in the same transaction as the\nchange it announces, and change data capture publishes it, so nothing is lost\nbetween saved and sent. Every consumer is idempotent, keyed by the order.\nMessages arrive at least once, and idempotency makes that effectively once.\n\nA distributed lock does not belong anywhere in this. Kleppmann's point is that a\nlock holder can pause past its lease — a garbage collection, a stalled virtual\nmachine — and keep writing, and the cure is a fencing token the store checks.\nBut a store that can check a token can check `available \u003e= :qty`, and then it\nnever needed the lock.\n\n## Pause the item, never oversell it\n\nConsistency and availability are chosen per question, not per system. \"Can I\nsell this unit?\" has to be consistent: when the owner of an item's stock cannot\nbe reached, that item stops selling. Everything else — the catalogue, the\n\"3 left\" badge, the order history — can stay available and be a little stale.\n\nThe quietest oversell comes from a failover. With asynchronous replication a\nprimary acknowledges a sale and dies before its replica has it; the replica is\npromoted, the sale is gone, and the unit is sold again. The fix is to make a\nsale durable on more than one machine before acknowledging it: a synchronous\nstandby in PostgreSQL (`synchronous_commit` with `synchronous_standby_names`),\nor a store built on Raft. That costs latency on every commit, which lengthens\nexactly the lock hold time the ceiling divides by, and batching and the gate\nbuy it back. Scope the pause, too: a failed shard should stop the items on it,\nnot the shop.\n\n## Proving it\n\nNone of this should be taken on trust, ours included.\n\n- **Watch the invariant.** Run `available + held + sold = total` on a schedule,\n  chart it, and page on any drift. Zero oversells is a service level objective\n  with an alert, not a hope.\n- **Race it on purpose.** Ten thousand concurrent buyers for a hundred units, in\n  the test suite, asserting exactly a hundred sold and no number below zero.\n- **Break things during the race.** Kill the primary, cut the network, pause a\n  process — Jepsen-style — and check the invariant afterwards.\n- **Load-test the rejections.** The sold-out path has to stay fast when it is\n  99% of the traffic, and the edge has to absorb it.\n- **Put the metrics on one dashboard:** lock waits, expired holds, and the\n  distance between the gate's count and the database's.\n\n## In what order\n\n1. Measure first. `pg_stat_statements` and the lock waits say whether the hot\n   row is the problem at all.\n2. Make the write one conditional statement.\n3. Shed load: a waiting room, a gate, a cached \"sold out\".\n4. Give inventory its own service and its own owner.\n5. Shard by `(tenant_id, item_id)`, and only if the first four were not enough.\n6. Add a hot-item mode, buckets or a single writer, for the sale you know is\n   coming.\n7. Regions last, with quotas.\n\n## What we would tell the next person\n\n- **Sharding answers a different question.** It spreads many rows, and an\n  oversold sale is one row. Find out which one you have before you build.\n- **Make the check and the decrement one statement.** Most of the distance\n  between fine and oversold is the time between reading a number and writing\n  it.\n- **Most of the traffic is the answer no.** Make rejection the cheapest thing\n  the system does.\n- **Pause before you oversell.** An item that stops selling for thirty seconds\n  is an apology. An item sold twice is a refund, a support queue and a customer\n  who remembers.\n- **Write the invariant down and watch it.** It is one line, and it is the only\n  thing that says the design works in production rather than in review.\n\n## Sources\n\n- [PostgreSQL: Read Committed isolation](https://www.postgresql.org/docs/current/transaction-iso.html#XACT-READ-COMMITTED)\n- [PostgreSQL: `FOR UPDATE` and `SKIP LOCKED`](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE)\n- [PostgreSQL: synchronous replication](https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION)\n- [Bailis et al., Coordination Avoidance in Database Systems](https://www.vldb.org/pvldb/vol8/p185-bailis.pdf)\n- [Balegas et al., Extending Eventually Consistent Cloud Databases for Enforcing Numeric Invariants](https://arxiv.org/abs/1503.09052)\n- [Martin Kleppmann: How to do distributed locking](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html)\n- [Martin Fowler: The LMAX Architecture](https://martinfowler.com/articles/lmax.html)\n- [DynamoDB: designing partition keys](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html)\n- [DynamoDB: partitions and data distribution](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.Partitions.html)\n- [microservices.io: Saga](https://microservices.io/patterns/data/saga.html)\n- [microservices.io: Transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html)\n- [Redis: replication](https://redis.io/docs/latest/operate/oss_and_stack/management/replication/)\n- [Jepsen](https://jepsen.io/)\n\n\u003chttps://engineer.company/notes/selling-limited-stock-without-overselling/\u003e\n","date_modified":"2026-10-18T22:40:12Z","description":"A flash sale oversells on one hot row, not on a big database. What stops it: a conditional write, holds that expire, and a saga.","formats":{"gemtext":"https://engineer.company/notes/selling-limited-stock-without-overselling/index.gmi","gopher":"https://engineer.company/notes/selling-limited-stock-without-overselling/gophermap","html":"https://engineer.company/notes/selling-limited-stock-without-overselling/","json":"https://engineer.company/notes/selling-limited-stock-without-overselling/index.json","markdown":"https://engineer.company/notes/selling-limited-stock-without-overselling/index.md","pdf":"https://engineer.company/notes/selling-limited-stock-without-overselling/index.pdf"},"kind":"page","language":"en","title":"Selling the last ticket exactly once","url":"https://engineer.company/notes/selling-limited-stock-without-overselling/","version":1}