An app that ate 41 MB a second, stopped

Situation. The application filled memory until the operating system killed it. The recorded incident reached 111 GB of compressed pages on a 36 GB machine, climbing at roughly 41 MB a second, and the log file for a single short session held 610,996 lines. A machine in that state is not slow, it is unusable — the kill arrives after the swap has already made everything else on the desktop stop responding.

Task. The growth had to be found rather than guessed at, and every unbounded path had to be given a limit, because one bounded queue next to three unbounded ones is not a fix.

Action. There were three multiplying causes and the multiplication is why it was so fast. The event stream from the core was consumed without any bound, so events arrived faster than the interface could apply them and the backlog was retained rather than dropped. Every subscriber received every event and filtered afterwards, so the cost of one event was multiplied by the number of listeners, and each listener’s filtering allocated. And logging was unbudgeted, so each event produced log lines — which is the compounding term, because the volume of logging was proportional to the volume of the thing going wrong. The fix addressed all three: bounded buffers with an explicit policy for what happens when they fill, subscription by event type so a listener is only woken for events it wants, and a rate budget on logging that collapses repeats rather than writing each one. A regression test drives a high event rate and asserts the memory ceiling holds, so the bound is a property of the build rather than a comment.

Result. Memory stays flat under sustained load and the same session that produced 610,996 log lines produces 1,411. The honest note is that the rate budget on logging discards information: when something goes wrong quickly now, the record of it is deliberately incomplete, and that is a trade made knowingly against the alternative of a machine that stops.