Backend Development6 min read

Background Jobs & Queues

Process background jobs with Bull/BullMQ: job queues, retry strategies, concurrency, scheduled jobs, and distributed worker patterns.

queuesworkersbackground-jobsbull

Background job queues handle tasks that shouldn't run synchronously in HTTP request handlers: sending emails, processing images, generating reports, or calling slow external APIs. BullMQ (backed by Redis) is the standard Node.js solution with durability, retries, and distributed workers.

Never do slow work in HTTP request handlers. Queue it and process asynchronously. Use BullMQ for reliable, durable queues. Configure exponential backoff for retries. Monitor job failures actively. Keep jobs idempotent.

Workflow

Set up BullMQ queues and workersBullMQ provides durable, Redis-backed queues that survive process restarts.
Design idempotent jobsJobs may run multiple times (due to retries or at-least-once delivery) - idempotency prevents duplicate side effects.
Configure retry strategiesTransient failures (API timeouts, DB blips) should be retried; permanent failures should fail fast.
Schedule recurring jobsCron-like scheduling replaces external cron jobs with managed, monitored queue workers.
Monitor queues with Bull BoardVisibility into queue depths, failure rates, and job details prevents surprise queue buildup.

Output Format

BullMQ Queue + Worker per job type, exponential backoff with attempt limit, `jobId` for deduplication, recurring jobs checked-before-adding, Bull Board behind auth middleware.

Examples

Worker with error handling: `const worker = new Worker("image-process", async (job: Job<ImageJob>) => { const { imageUrl, userId } = job.data; job.updateProgress(10); const processed = await processImage(imageUrl); job.updateProgress(80); await db.images.update(userId, { processed }); job.updateProgress(100) }, { connection, concurrency: 3, limiter: { max: 10, duration: 1000 } })`

Idempotent job: `await queue.add("send-sms", { phone, message, orderId }, { jobId: `sms-${orderId}-${phone}`, attempts: 3, backoff: { type: "exponential", delay: 2000 } })`

Queue flow (job chain): `const flow = new FlowProducer({ connection }); await flow.add({ name: "process", queueName: "main", children: [{ name: "validate", queueName: "validation" }, { name: "notify", queueName: "notifications" }] })`

Edge Cases & Gotchas

Redis failure: BullMQ requires Redis; implement graceful degradation if Redis is down (log the failure, optionally retry synchronously for critical tasks)
Worker crashes: if a worker crashes mid-job, the job stays in "active" state; BullMQ has a lock timeout (`lockDuration`) after which the job is returned to "waiting" and retried
Fan-out: one job producing many child jobs (e.g., send email to 10,000 users) - use BullMQ FlowProducer or add child jobs in the parent worker; avoid adding millions of jobs at once (use cursor-based batching)

When to Break These Rules

For very simple scheduled tasks with low reliability requirements, a basic cron job (system cron or cloud scheduler) is simpler than a full job queue setup.

Reference Files

BullMQ docs: https://docs.bullmq.io/
Bull Board: https://github.com/felixmosh/bull-board
BullMQ patterns: https://docs.bullmq.io/patterns
Temporal (workflow orchestration alternative): https://temporal.io/