Queue

work left in order for something else to pick up

What it is

A queue is a holding place between two programs: one side puts messages or jobs in, the other takes them out when it is ready, usually oldest first. Most systems run several queues, one per kind of work. The program that adds work is the producer, the program that takes it is the consumer, and whatever stores the messages in between is the broker, which can be a dedicated server or a table in a database the system already has.

The arrangement lets the two sides stop waiting for each other. A web request that needs an email sent adds a message and answers the user at once; a worker picks the message up moments later, and if a hundred arrive together, it drains them at its own pace. A burst becomes a backlog, and a dependency becomes a delay.

Two ideas describe what a queue promises about delivery. The first is acknowledgement: a consumer tells the broker when it has finished with a message, and only then is the message removed. In the AMQP 0-9-1 model, which RabbitMQ implements, a message whose consumer dies before acknowledging goes to another consumer. The second is durability: a durable queue of persistent messages survives a broker restart, and a queue held only in memory does not.

Three neighbouring terms are worth keeping apart. A message queue carries data between services. A job queue carries units of work for background workers, each with its arguments, a state and a count of attempts. A log, the model behind Apache Kafka, looks like a queue from the outside but keeps events after they have been read, for a retention period chosen per topic, so several readers can replay the same history. Reading from a queue removes the message; reading from a log only moves the reader’s position.

A queue need not be a separate product at all. PostgreSQL can play the part with a table of jobs and a query that claims the next row while stepping over rows another worker has already locked. The manual names queue-like tables as the reason the SKIP LOCKED clause exists:

WITH next AS (
  SELECT id
    FROM jobs
   WHERE state = 'available'
   ORDER BY id
   LIMIT 1
     FOR UPDATE SKIP LOCKED
)
UPDATE jobs
   SET state = 'running',
       attempt = attempt + 1
  FROM next
 WHERE jobs.id = next.id
RETURNING jobs.id, jobs.args;

Why it matters

A queue decides what a user waits for. Work placed on one happens after the response has gone, so a page that triggers an email, a PDF, a search reindex and a recalculation can answer in the time it takes to write four rows, and the four slow tasks finish behind it. For a buyer, that is the difference between a checkout that responds in a few hundred milliseconds and one that responds whenever the mail server is ready.

It also decides what survives a failure. Work started in a background thread lives in that process’s memory, and a deploy that restarts the process takes the work with it, leaving no trace that it was ever due. Work on a durable queue is a record instead: it can be retried, inspected and counted.

It separates two speeds, too. Producers move at the speed of incoming traffic; consumers move at the speed of the slowest service they call. A queue between them absorbs the difference, and its depth becomes one of the most truthful numbers a system can report. A queue that has grown for an hour says the consumers are losing ground, whatever the processor graph suggests.

The costs arrive with the benefits. Work now sits out of sight between being sent and being done. Delivery is almost always at least once, so a consumer has to tolerate seeing a message twice. Ordering is weaker than it looks. And a dedicated broker is one more server to run, secure, watch and upgrade.

What it looks like in practice

Sending mail, rebuilding a search index, generating documents and recalculating standings were all moved off the request path on one platform. Work handed to a goroutine vanishes when the process restarts in the middle of a deploy, so the work went onto a River job queue kept in PostgreSQL. The database was already the system of record, which makes enqueuing a job part of the same transaction as the data it concerns. What the platform gained is retries and a history someone can open and read; the stated limit is that a queue inside the database will not scale forever.

In a macOS application built over a C library, the queue was there to keep order. The library’s blocking event loop emits events in a sequence that matters, because an edit has to arrive after the message it edits. Starting one task per event lets the runtime schedule them in any order, so an edit can overtake the message it refers to. The design that shipped has one producer thread, one consumer and one queue between them, followed by a single hop onto the main actor. Order holds because there is exactly one path through. The price is parallelism: every event passes through that one consumer.

A queue can also do harm. In one application the event stream was consumed without any bound: events arrived faster than the interface could apply them, the backlog was kept, and as one of three causes it helped memory climb at about 41 MB a second and push compressed pages to 111 GB on a machine with 36 GB. Every unbounded path was given a limit, with an explicit policy for a full buffer, because one bounded queue next to three unbounded ones is not a fix. A regression test now asserts the memory ceiling.

Where it goes wrong

Queue trouble surfaces late and somewhere else, because the work is out of sight by design.

The backlog grows until memory or disk gives out. A queue with no limit accepts work faster than it is done and keeps the difference. Give every queue a maximum length and decide ahead of time what happens at that length: slow the producer, refuse new work, or drop the oldest. Then watch two numbers, the depth and the age of the oldest message, because a short queue of very old messages is as bad as a long one.

The same message is processed twice. A consumer finishes the work and dies before it acknowledges; the broker, correctly, delivers the message again. RabbitMQ’s documentation tells consumers to expect redeliveries and to be written with idempotence in mind. Give each message an identifier and record the identifiers already handled, in the same transaction as the work itself. A note on this site about selling limited stock without overselling builds its writer around exactly that assumption.

Messages arrive in a different order from the one they were sent in. Parallel consumers, prefetching, and retries that put a failed message behind newer ones all reorder work. When order matters, route everything about one account or one order through the same partition or the same consumer. Kafka does this by key, at the cost that one busy key moves only as fast as one consumer.

One bad message blocks or loops forever. A message whose payload crashes the consumer every time is requeued, delivered again, and crashes it again. Cap the attempts, and move a message that has used them all to a separate place where a person can look at it. RabbitMQ calls this dead-lettering and applies it to rejected messages, expired ones and messages that would overflow a length limit.

Work disappears on restart. A channel, a list in memory or a background thread is a queue that keeps nothing across a restart. In AMQP, a durable queue alone is not enough either: the messages must be published as persistent too. Decide per kind of work whether losing it is acceptable.

The database and the queue disagree. A service writes a row, then publishes a message, and crashes between the two; now one exists without the other. Either keep the queue in the same database, so that a job becomes visible to workers only when its transaction commits, which is how River behaves, or write the message into an outbox table in that transaction and publish it from there.

Further reading

Glossary