This article is published in English.
Building Reliable Background Job Systems with BullMQ and Redis
Learn how to design resilient Node.js background job pipelines with BullMQ and Redis, covering retries, concurrency, idempotency, and monitoring.
Sending a confirmation email, generating a report, processing a payment — plenty of backend work doesn't need to finish before you respond to a user. This guide walks through building dependable background job systems using BullMQ paired with Redis.
Picture a backend where nearly every task happens directly inside the HTTP request cycle. Need to send an email? Handle it right there. Need to generate a PDF? Same thing. Need to crunch some background data? Do it inline too.
This approach works fine at first. Then it stops working.
The API begins to slow down. Requests start timing out. And if some external service goes down, the whole request can fail along with it.
That's the point where background jobs start to make sense.
Rather than forcing the API to complete every step before answering, you can push the work into a queue and let a dedicated worker handle it separately.
A solid option for this in the Node.js ecosystem is BullMQ, backed by Redis for storage. Here's how the pieces fit together.
1. What Is a Background Job?
A background job is any unit of work that doesn't have to run synchronously as part of an HTTP request.
Consider a typical signup flow. When someone creates an account, the API might need to:
- Create the user record
- Send a welcome email
- Generate a welcome PDF
- Send a notification
- Update some other downstream system
You could try to run all of this inline before responding:
Client
↓
API
↓
Create User
↓
Send Email
↓
Generate PDF
↓
Send Notification
↓
Response
But that forces the user to wait for every one of those steps to finish.
A better approach looks like this instead:
Client
↓
API
↓
Create User
↓
Add Job to Queue
↓
Response
And separately:
Queue
↓
Worker
↓
Send Email
↓
Done
Because the API no longer has to complete every task before replying, it responds much faster.
2. Why Do We Need a Queue?
Suppose sending an email takes 1 second, generating a PDF takes 2 seconds, and calling another API takes 1 second. Your endpoint could end up stalling for several seconds before it sends a response — a poor experience for the user.
Worse, what happens if the email provider is unreachable? The request could fail even though creating the user actually succeeded. That's an unnecessary dependency between two unrelated concerns.
A queue breaks that coupling:
┌──────────────┐
│ Node API │
└──────┬───────┘
↓
Add Job
↓
┌──────────────┐
│ Redis │
│ Queue │
└──────┬───────┘
↓
┌──────────────┐
│ Worker │
└──────┬───────┘
↓
Email / PDF / API / etc.
With this split, the API and the background task each own a distinct responsibility.
3. What Is BullMQ?
BullMQ is a queue library for Node.js that relies on Redis to store and coordinate jobs. Its architecture, at a high level, looks like this:
Producer
↓
Queue
↓
Worker
↓
Job Processing
The producer is what creates jobs. The queue stores them. The worker is what actually processes them.
For instance:
await emailQueue.add("welcome-email", {
userId: user.id,
email: user.email
});
Essentially, the API is saying:
"Here's some work that needs to happen."
It doesn't need to carry out that work itself.
4. Creating a Queue
Here's what a minimal BullMQ queue setup looks like:
import { Queue } from "bullmq";
const connection = {
host: "localhost",
port: 6379
};const emailQueue = new Queue("email", {
connection
});
From there, you can push jobs onto it:
await emailQueue.add("welcome-email", {
userId: "123",
email: "user@example.com"
});
Redis handles storing all the queue-related state behind the scenes. Conceptually, you can think of it like this:
email queue
Job 1
Job 2
Job 3
Job 4
Job 5
The worker then picks up and consumes these jobs.
5. Creating a Worker
The worker is the piece that actually performs the work:
import { Worker } from "bullmq";
const worker = new Worker(
"email",
async (job) => {
console.log("Processing:", job.name); await sendWelcomeEmail(
job.data.email
);
},
{
connection
}
);
Putting it together, the flow now looks like this:
API
↓
emailQueue.add()
↓
Redis
↓
Worker
↓
sendWelcomeEmail()
The API doesn't have to wait around for the email to finish sending — and that's the core benefit background jobs provide.
6. What Happens When a Job Fails?
This is precisely where a queue starts to show its real advantage over a plain service call.
Picture this setup:
API
↓
Email Service
↓
ERROR
When you call an API directly, you're forced to make an immediate decision about failure.
A queue gives you another option: the job can simply be retried.
Here's an example:
await emailQueue.add(
"welcome-email",
{
email: "user@example.com"
},
{
attempts: 3
}
);
With this configuration, the job is allowed several attempts before giving up.
Visually, the flow looks like this:
Attempt 1
↓
Failed
↓
Attempt 2
↓
Failed
↓
Attempt 3
↓
Success
This pattern is extremely valuable when you're dealing with flaky third-party services.
That said, retries shouldn't be unlimited or careless.
Avoid a setup where a job keeps retrying forever with no boundary.
7. Retry With Backoff
Suppose an external service goes down temporarily.
What you want to avoid is something like this:
FAIL
RETRY IMMEDIATELY
FAIL
RETRY IMMEDIATELY
FAIL
RETRY IMMEDIATELY
Hammering a struggling service with instant retries can actually make things worse.
The fix is to introduce backoff between attempts.
For example:
await emailQueue.add(
"welcome-email",
{
email: "user@example.com"
},
{
attempts: 5,
backoff: {
type: "exponential",
delay: 5000
}
}
);
Here's what that looks like conceptually:
Attempt 1 → Fail
↓
5 sec
↓
Attempt 2 → Fail
↓
10 sec
↓
Attempt 3 → Fail
↓
20 sec
↓
Attempt 4 → Success
The precise timing depends on how you configure the retry and backoff strategy.
But the underlying idea stays the same:
Give temporary failures room to recover before trying again.
8. Delayed Jobs
Not every job needs to run the moment it's created.
For instance:
Send a reminder 24 hours after signup.
BullMQ lets you schedule a job to run later.
await emailQueue.add(
"reminder",
{
userId: "123"
},
{
delay: 24 * 60 * 60 * 1000
}
);
Conceptually:
Create Job
↓
Wait 24 hours
↓
Worker processes job
This pattern shows up in situations like:
- Reminder emails
- Scheduled notifications
- Trial expiration
- Payment reminders
- Follow-up messages
9. Multiple Workers
Now picture a system receiving thousands of jobs per minute.
A single worker process may not keep up.
You can scale out by running several workers at once:
Redis Queue
↓
┌──────────┼──────────┐
↓ ↓ ↓
Worker 1 Worker 2 Worker 3
↓ ↓ ↓
Jobs Jobs Jobs
Each one pulls jobs from the queue independently.
For example, given:
1000 email jobs
You might see something like:
Worker 1 → Job 1, 4, 7...
Worker 2 → Job 2, 5, 8...
Worker 3 → Job 3, 6, 9...
Adding more workers is one way to boost throughput.
But be careful:
Throwing more workers at the problem isn't automatically a win.
Your database, your email provider, your CPU, your memory, and any downstream service all have their own capacity limits.
10. Concurrency
Beyond running multiple worker processes, BullMQ also lets you configure how many jobs a single worker handles at the same time.
For example:
const worker = new Worker(
"email",
async (job) => {
await sendEmail(job.data.email);
},
{
connection,
concurrency: 5
}
);
This allows one worker to process several jobs in parallel.
Conceptually:
Worker
├── Job 1
├── Job 2
├── Job 3
├── Job 4
└── Job 5
Higher concurrency can raise throughput.
But don't just crank concurrency up to 100 without thinking it through.
If each job hits your database, high concurrency could easily overload it.
Concurrency settings should be tuned to match what your workload can actually support.
11. Rate Limiting
Sometimes the bottleneck isn't inside your own system at all — it's the third-party service you depend on.
Say your email provider caps you at a fixed number of requests per second.
If you suddenly have:
10,000 jobs
you don't want to fire them all off at once.
A queue can throttle the pace at which jobs get processed.
The resulting architecture looks like:
10,000 Jobs
↓
Queue
↓
Rate Limit
↓
Worker
↓
External API
This is far safer than blasting thousands of simultaneous requests at a provider.
12. Job Idempotency Matters
This next concept is one of the most critical ideas in background job processing.
Consider a payment-processing job:
Process Payment
The worker runs it.
The payment goes through successfully.
But right before the worker marks it as complete, the process crashes.
The queue, doing exactly what it's designed to do, retries the job.
Without safeguards, you could end up charging the customer a second time.
That's a real and costly problem.
To prevent this, jobs should be written to be idempotent wherever it's feasible.
In practice, that means running the same job twice shouldn't create an unintended duplicate side effect.
One common approach is to key off a unique payment reference:
payment:order_123
Then, before doing any work, check:
Has this payment already been completed?
↓
Yes → Don't charge again
↓
No → Process payment
BullMQ itself has no built-in mechanism for this.
It's on your application code to enforce idempotency.
13. Failed Jobs Need a Strategy
Not all failures are equal, and not all of them are worth retrying.
Consider a few examples:
Invalid email
Invalid user ID
Missing database record
Invalid payment information
Running these jobs again five times isn't going to fix anything.
It helps to separate failures into two categories:
Temporary failures
These include things like:
- A network timeout
- A dependency that's briefly unavailable
- A dropped database connection
These are the kinds of problems where trying again later actually makes sense.
Permanent failures
These include things like:
- Bad input data
- A referenced resource that no longer exists
- A violation of a business rule
For these, retrying is pointless — the job needs to go straight into some kind of failure-handling path instead.
A well-designed queue setup doesn't just follow a blanket rule of:
Retry everything
Instead, it follows a more deliberate flow:
Understand why it failed
↓
Temporary?
/ \
YES NO
↓ ↓
Retry Handle failure
14. Dead-Letter / Failed Job Handling
No matter how careful you are, some jobs will fail in a way that can't be fixed by retrying. You need visibility into those jobs so they don't just disappear.
For example, you might end up with something like:
Failed Jobs
──────────────
Job 101 → Email invalid
Job 102 → Payment failed
Job 103 → API timeout
Once you can see these failures, you have options:
- Record the failure for later review
- Notify your team
- Give someone the ability to retry manually
- Correct whatever bad data caused it
- Route the job into a dedicated failure-handling workflow
Exactly how you build this depends on your system's needs. What matters most is a single principle:
Failed work should never vanish without a trace.
15. Queue vs Cron Job
It's easy to mix these two up, but they solve different problems.
A cron job's job is to say:
"Run this task at a particular time."
A queue's job is to say:
"Process this unit of work."
In practice, these two tools often work well together. For instance:
Cron
↓
Find users whose trial expires today
↓
Create jobs
↓
Queue
↓
Workers
↓
Send emails
This keeps scheduling logic separate from processing logic. That's usually a cleaner design than having a single cron process try to do all the work itself.
16. Queue Events and Monitoring
Once you're running this in production, you need visibility into what's actually happening inside the queue.
Metrics worth tracking include:
- Jobs waiting to be picked up
- Jobs currently being processed
- Jobs that completed successfully
- Jobs that failed
- How long processing takes
- How many retries are happening
- Overall queue size
Picture a dashboard that suddenly shows something like this:
Waiting Jobs
Normal: 50
Current: 25,000
That kind of jump is a warning sign. It could mean:
- Your workers have stopped running
- An external API has slowed down
- Your database is under heavy load
- Traffic has spiked
- A recent deployment shipped a bug
If you're not monitoring your queue, these problems can pile up invisibly until users start noticing something is wrong.
17. Don't Put Everything Into a Queue
Having BullMQ available doesn't mean every single operation belongs in a background job.
Take something like:
GET /profile
Here, the user is waiting for their profile data right away. Deferring that into a background queue wouldn't make sense — it would just add unnecessary delay.
A queue makes sense when:
- The work takes a while to complete
- The work can be done asynchronously
- The work might need to be retried
- The work is resource-heavy
- The work depends on external services that aren't fully reliable
- The result doesn't need to be part of the immediate response
A useful question to ask is:
Does the user actually need this result before you send back the HTTP response?
If not, it's worth considering moving that work into a background job.
18. A Production-Style Architecture
Bringing all of this together, a typical setup looks like this:
Client
↓
Node.js API
↓
┌──────┴──────┐
↓ ↓
PostgreSQL Redis
↓
Queue
↓
┌──────────┼──────────┐
↓ ↓ ↓
Worker 1 Worker 2 Worker 3
↓ ↓ ↓
Email PDF Notifications
The API layer deals with whatever needs to happen immediately. PostgreSQL (or your database of choice) holds your durable business data. Redis supports the queue infrastructure and other short-lived workloads where it's a good fit. Workers take care of everything that can happen asynchronously.
Splitting responsibilities this way makes the whole system considerably easier to scale.
19. Mistakes Worth Avoiding
Mistake 1: Doing everything inside the HTTP request
This leads to APIs that are both slow and fragile.
Mistake 2: Retrying without limits
Some failures simply aren't going to resolve themselves no matter how many times you try.
Mistake 3: Skipping idempotency
If a job runs twice, it can trigger duplicate side effects you didn't intend.
Mistake 4: Allowing unlimited concurrency
Without limits, you risk overwhelming the systems your jobs depend on.
Mistake 5: Skipping monitoring
A queue that keeps growing unchecked is an operational issue waiting to surface.
Mistake 6: Using Redis as your system of record
Queue state and core business data serve different purposes and shouldn't be conflated.
Mistake 7: Making everything asynchronous
Some operations genuinely need to complete before you send a response back.
20. A Better Mental Model
Before understanding queues, the natural instinct is to think of request handling like this:
Request
↓
Do everything
↓
Response
A more useful model looks like this instead:
Request
↓
Do what must happen immediately
↓
Queue what can happen later
↓
Response
Followed by:
Queue
↓
Worker
↓
Process
↓
Retry if appropriate
↓
Complete / Fail
That shift — separating what must happen now from what can happen later — is the core idea behind all of this.
Final Takeaway
BullMQ isn't useful simply because it's a widely used Node.js library. It's useful because background job processing addresses a genuine architectural need.
If a piece of work is:
- Slow
- Something that can be retried
- Something that doesn't need to block the response
- Dependent on an external service
- Resource-intensive
then it probably shouldn't sit inside your HTTP request cycle.
A queue gives that work somewhere to live. Redis supplies the underlying infrastructure. BullMQ handles job management. Workers carry out the actual processing. Retries take care of temporary failures. Concurrency settings keep throughput under control. Monitoring lets you know when something's off. And thoughtful application-level design ensures jobs can safely run more than once when that becomes necessary.
The core lesson here is this:
Not everything has to be resolved within the request-response cycle.
Sometimes the right response to send back is simply:
"I've accepted the work. We'll take care of the rest."