Debugging a Node.js Memory Leak That Only Showed Up in Production
Our API's memory usage climbed steadily for about six hours after every deploy, then the process restarted itself once it hit the container's memory limit, quietly, with no errors in the logs pointing at anything obvious. It never happened locally, never happened in staging, only in production under real traffic, which meant I couldn't just attach a debugger and step through the problem the way I usually would.
Reproducing it locally first, since production debugging is miserable
I wrote a small script using autocannon to hammer our staging environment with sustained load for an hour, watching memory climb in a way it never had during casual manual testing. This confirmed the leak was load-dependent, not deploy-dependent, which narrowed the search considerably and meant I could finally reproduce it on my own machine instead of guessing at production behavior.
Taking heap snapshots, and what actually showed up
I ran the process with the --inspect flag and connected Chrome DevTools' memory profiler, taking a snapshot at startup and another after ten minutes of sustained load. Comparing the two in the "Comparison" view showed a huge, steadily growing count of retained objects, all instances of a class I recognized immediately, our request logger's context object.
The actual bug, an event listener that never got removed
Every incoming request attached a new listener to a shared EventEmitter used for cross-cutting logging, meant to fire once and clean itself up, but a refactor months earlier had swapped emitter.once() for emitter.on() without anyone noticing, meaning every single request added a listener that never got removed. Under low local traffic this leaked slowly enough to never matter. Under real production load, it accumulated fast enough to crash the process within hours.
// before, the actual bug
emitter.on('request:complete', (ctx) => {
logger.flush(ctx);
});
// after, the fix
emitter.once('request:complete', (ctx) => {
logger.flush(ctx);
});
Confirming the fix before shipping it
I re-ran the same autocannon load test against a branch with the one-line fix, watching the heap snapshot comparison this time, and the retained object count stayed flat across the full hour instead of climbing, the clearest possible confirmation that this specific listener was the actual leak rather than one of several contributing factors.
Why our existing monitoring didn't catch this sooner
We had memory alerts configured, but they only fired on a hard threshold breach, not on a sustained upward trend, meaning the alert only ever fired seconds before the container restarted, giving us zero lead time to actually investigate before the process recycled itself and memory dropped back to baseline, erasing the evidence.
Adding a trend-based alert instead of a threshold-based one
I added a second alert specifically watching for memory growth rate over a rolling thirty-minute window rather than an absolute value, which would have caught this leak within the first hour after deploy instead of letting it run silently for six. This felt like a genuinely small addition, but it's caught two smaller leaks since that we'd have otherwise only noticed through a restart loop again.
A lint rule to catch the actual root cause going forward
Since the underlying mistake was a single-character difference between .once() and .on() that a code reviewer could easily miss, I added a custom ESLint rule flagging any EventEmitter listener attached inside a request handler without an explicit cleanup call nearby, catching the pattern automatically before it ever reaches review rather than relying purely on human attention to spot it again.
Setting a more conservative memory limit as an additional safety net
Beyond fixing the actual leak, I lowered the container's memory limit closer to what the app genuinely needs under normal load, specifically so any future leak would trigger a restart, and the trend-based alert, well before it had six hours to run wild the way this one did. A tighter limit forces problems to surface sooner rather than giving a slow leak room to grow quietly for most of a day.
Checking whether other services shared the same EventEmitter pattern
Once I understood the actual root cause, I searched the rest of our codebase for the same .on() versus .once() pattern attached inside any per-request handler, and found one other service with an identical, if smaller-scale, version of the same mistake, fixed before it ever grew large enough to trigger its own restart loop and cost someone else the same debugging session.
Using --expose-gc to force collection during comparison, and why that mattered
My first few heap snapshot comparisons were noisy, showing growth that partly reflected objects simply waiting for the next garbage collection cycle rather than genuinely leaked memory. Running the process with --expose-gc and forcing a manual collection immediately before each snapshot removed that noise, leaving only genuinely retained objects in the comparison rather than a mix of real leaks and normal, temporary garbage collection lag.
What I'd tell someone chasing a production-only leak
The instinct to debug directly in production is understandable when something only happens there, but building a reliable local reproduction first, even a crude load test script, turned a days-long mystery into an hour of heap snapshot comparison once I actually had a repeatable failure in front of me. Chasing intermittent production behavior directly, without first isolating a reproducible case, is exactly how these investigations stretch on for weeks instead of hours.
