Debugging a Memory Leak in Node.js: What Actually Found It
Our order processing service was restarting itself roughly every eighteen hours, silently, because Kubernetes was killing pods that exceeded their memory limit and rescheduling them before anyone noticed. I only found out this was happening because a teammate mentioned in passing that our error rate had a strange sawtooth pattern in our dashboards, climbing steadily then dropping to zero right before climbing again, which is exactly what a memory leak forcing periodic restarts looks like on a graph.
My first instinct was wrong and cost me two days
I assumed the leak was in a large in-memory cache we'd added for frequently accessed product data, since that felt like the obvious suspect, a growing collection of objects with no eviction policy. I spent two days adding size limits and TTLs to that cache, redeployed, and the memory graph looked exactly the same, climbing steadily toward the limit on the same roughly eighteen-hour cycle. The cache wasn't the problem at all.
Taking actual heap snapshots instead of guessing
I stopped guessing and used Node's built-in --inspect flag with Chrome DevTools to capture heap snapshots at two different points, right after a fresh restart and again about twelve hours later, then used the comparison view to see exactly what object types had grown between the two snapshots.
node --inspect=0.0.0.0:9229 server.js
# then connect via chrome://inspect
# Memory tab > Take Heap Snapshot
# compare "Allocation instances" between snapshotsWhat the comparison actually showed
The retained object count that had grown the most wasn't application data at all, it was EventEmitter listener references, specifically tied to a database connection pool. We were creating a new event listener on every single database query to log slow queries, and never removing that listener once the query finished, meaning every query we'd ever run since the last restart still had a dangling listener attached to the pool's event emitter.
The actual line of code, once I found it, was almost embarrassing
Buried in a query wrapper function, someone had added pool.on('slow-query', callback) inside the function that ran on every query, rather than registering that listener once at application startup. Every single database call added a new permanent listener that never got cleaned up, thousands of them accumulating over the life of the process.
// the leak
async function runQuery(sql, params) {
pool.on('slow-query', (info) => logSlowQuery(info)); // new listener every call
return pool.query(sql, params);
}
// the fix, registered once outside the function
pool.on('slow-query', (info) => logSlowQuery(info));
async function runQuery(sql, params) {
return pool.query(sql, params);
}Why this specific bug was so hard to spot in code review
The function looked completely reasonable in isolation, and nobody reviewing that pull request months earlier would have known to ask whether it ran once or on every call, since the listener registration was syntactically identical either way. This is the kind of bug that's genuinely difficult to catch by reading code and much easier to catch by actually measuring what the running process is doing.
Setting up a warning system so we'd catch the next one faster
After fixing this, I added a simple check that logs a warning if any EventEmitter in our process exceeds Node's default max listener count, which is normally just a console warning easy to ignore in logs. We now treat that specific warning as a paging alert in our monitoring, since it's a genuinely reliable early signal for exactly this class of bug.
process.on('warning', (warning) => {
if (warning.name === 'MaxListenersExceededWarning') {
alerting.page('Possible listener leak detected', warning);
}
});What actually fixed the eighteen-hour restart cycle
Memory usage flattened out completely after deploying the fix, sitting stable instead of climbing, and we haven't seen an unexplained pod restart on that service since. The whole experience taught me something I now repeat to anyone debugging a memory leak, don't trust your first instinct about what's leaking, take an actual heap snapshot comparison before touching any code, because my two wasted days chasing the cache were pure guesswork that a five-minute snapshot comparison would have avoided entirely.
