The service generated PDF reports for a logistics client. Memory rose steadily from boot, and Kubernetes killed it around the six-hour mark.
Three snapshots, one comparison
Take a heap snapshot after boot, another after 1,000 requests, and a third after 2,000. Compare the second and third. Whatever grows by the same amount between them is your leak.
The comparison pointed at closures retained by an EventEmitter. Each request registered a progress listener with on() and never removed it.
// before
jobs.on("progress", report);
// after
jobs.once("done:" + id, report);Memory now sits flat at 180 MB. We also added a check to CI that fails if any emitter passes 50 listeners.
Why heap snapshots beat guessing
Our first instinct was to blame the cache, because caches are the usual suspect and the service used one. A snapshot comparison doesn't care what looks suspicious — it only shows what actually grew between two points under identical load. That's what pointed away from the cache (which was evicting correctly) and at the EventEmitter (which wasn't).
The general pattern with EventEmitters
.on() keeps a listener registered forever unless something explicitly calls .off() or removeListener(). Any code that registers a listener per request — a progress handler, a logger, a socket subscription — is a leak by default unless it's scoped to fire once or is cleaned up when the request ends. .once() is the easy fix when the listener really only needs to fire a single time; otherwise the cleanup has to happen explicitly, usually in a finally block.
Making it a regression class, not a one-off fix
Finding one leak tells you the shape of the bug, not that it won't happen again in a different handler. The CI check that fails above 50 listeners on any single emitter turns "we found a leak" into "we can't reintroduce this class of leak without a build failing first."