Building a Job Queue With BullMQ: What I Learned Running It in Production
Every "hello world" job queue tutorial ends right after the first job completes successfully, which is the easy part. Running BullMQ 5 against real traffic for the better part of a year, with jobs that call flaky third-party APIs and occasionally need to run twice due to network retries, is where the actual lessons live, mostly around what happens when things go wrong rather than when they go right.
Queue, Worker, and QueueEvents Are Genuinely Separate Concerns
Coming from Bull, the older library, the split into three distinct classes, Queue for adding jobs, Worker for processing them, QueueEvents for listening to lifecycle events from a separate process, felt like unnecessary ceremony at first. It stopped feeling that way once I needed a monitoring service subscribing to job completion events without also being a worker that processes jobs, something Bull's more merged API made awkward. The separation maps cleanly onto how a real system is actually structured: producers, workers, and observers are different responsibilities that shouldn't have to live in the same process.
import { Queue, Worker, QueueEvents } from 'bullmq';
import IORedis from 'ioredis';
const connection = new IORedis({ maxRetriesPerRequest: null });
const emailQueue = new Queue('email', { connection });
const worker = new Worker('email', async (job) => {
await sendEmail(job.data);
}, { connection, concurrency: 5 });
worker.on('failed', (job, err) => {
console.error(`job ${job.id} failed:`, err.message);
});Idempotent Job Handlers Aren't Optional
A job can run more than once, a worker crashing mid-execution after the side effect happened but before BullMQ marked it complete is the classic case, and designing every job handler to be safe to run twice was the single habit that prevented the most production incidents. For the email queue specifically, this meant checking an "already sent" flag keyed by a deterministic id derived from the job's actual business data before sending, not trusting that BullMQ's own retry accounting alone would prevent a duplicate send.
The Concurrency and Rate Limiter Options Solve Different Problems
concurrency controls how many jobs one worker processes in parallel; limiter controls the maximum job throughput over a time window regardless of concurrency. I conflated these early on and set a high concurrency assuming it alone would control load on a downstream SMTP provider with its own rate limits, which just meant hitting that provider's rate limit externally instead. Pairing a reasonable concurrency with an explicit limiter, max jobs per duration, targeting the downstream service's actual documented limit is what stopped us from getting throttled or blocked by that provider.
const worker = new Worker('email', handler, {
connection,
concurrency: 5,
limiter: { max: 100, duration: 60_000 }, // 100 jobs per minute, matching provider limits
});Dead Letter Handling Instead of Silent Failure After Retries
A job that exhausts its configured retry attempts just sits in BullMQ's failed set by default, invisible unless someone happens to check the queue dashboard. Adding a failed listener that, once a job's attemptsMade equals its max attempts, moves relevant details into a separate dead-letter table and fires an alert turned "silently failed forever" into "someone gets paged and can manually inspect or replay it," which is the actual point of having retries in the first place rather than just delaying the same silent failure.
Running Workers as a Separate Process From the API
An early mistake was starting the worker inside the same Node process as the Express API, for deployment simplicity. A worker that throws an uncaught exception processing a bad job payload took the whole API down with it, since they shared a process and an unhandled worker error crashed the same event loop the API needed. Splitting workers into their own deployable process, scaled and restarted independently of the API, isolated that blast radius entirely; a bad job now only affects job processing, not user-facing requests.
Watching Redis Memory, Not Just Queue Depth
Completed jobs stay in Redis unless explicitly cleaned up, and a queue doing tens of thousands of jobs a day accumulates gigabytes of completed-job data within weeks if nothing prunes it. Setting removeOnComplete with a count limit, keep the last thousand completed jobs for debugging visibility, and removeOnFailed with a longer retention since failures need more time to investigate, kept Redis memory bounded without losing the recent history that's actually useful when something goes wrong.
await emailQueue.add('welcome', data, {
attempts: 3,
backoff: { type: 'exponential', delay: 2000 },
removeOnComplete: { count: 1000 },
removeOnFail: { count: 5000 },
});Flow Producers for Jobs That Depend on Other Jobs
A report-generation pipeline needed several sub-jobs, fetch data, transform it, render a PDF, to complete in order with the final step depending on all the others finishing successfully. Manually chaining these by having each job enqueue the next one worked but scattered the pipeline's actual structure across multiple handler files. BullMQ's FlowProducer, which defines parent-child job dependency trees explicitly, made that pipeline's shape visible in one place instead of implicit in a chain of enqueue calls buried inside handlers.
Final Verdict
BullMQ handles the mechanics of a reliable job queue well out of the box, retries, backoff, dependency flows, but the lessons that actually mattered in production were about designing for the failure modes the library exposes rather than hides: idempotent handlers for jobs that run twice, explicit rate limiting separate from concurrency, dead-letter visibility instead of silent failure, and process isolation so a bad job can't take down the API next to it.
