Inside a Multi-Provider Notification Service: Queues, Fallbacks, Trust

Sending an email or SMS looks like a one-line job: call the provider's API and move on. In production that single send() call quietly becomes a distributed systems problem, because providers time out, rate-limit, go offline, deliver out of order, and retries can produce duplicates. This article dissects the architecture of an open-source, multi-tenant notification service built on Node.js, BullMQ and Redis, covering seven areas: secrets, provider failures, queue design, ordering, deliverability, tenant fairness and webhook verification. For each one you will see the decision, the reasoning behind it and the limitation it knowingly accepts, so you can reuse the reasoning in your own systems.

Two kinds of secrets, two storage strategies

A notification platform holds several credentials, and the deciding question for each is simple: will the system ever need the original value back?

Provider keys: encrypt, because they must be recovered

Customers bring their own email provider API keys, and the platform must present those keys to the provider on every send. That rules out hashing; the value has to be recoverable. The design validates each key against its provider first, then encrypts it with AES-256-GCM before storing it. GCM is an authenticated mode, so besides confidentiality it detects tampering: a ciphertext that has been altered fails to decrypt instead of producing a plausible but wrong credential.

The plaintext never touches the database, logs or caches. It is decrypted in exactly two places: to render a masked preview in the dashboard, and inside the send worker right before the outbound provider request. Narrowing decryption to a couple of well-defined code paths makes accidental leakage far easier to audit.

Platform-issued API keys: hash, because they only need verifying

The API keys the platform issues to its own customers follow the opposite logic. The system never has to recover them, only to confirm that a presented key is genuine. So the key is displayed once at creation, and only its SHA-256 hash is persisted. On each request, the incoming key is hashed and compared with the stored digest. A lost key cannot be retrieved and must be regenerated.

A fast hash like SHA-256 is appropriate here because the keys are long random strings with high entropy. The same choice would be wrong for human passwords, which need a deliberately slow algorithm such as bcrypt, scrypt or Argon2 to resist guessing.

Webhook signing secrets: somewhere in between

The secret used to sign outgoing webhooks is generated by the platform, like an API key, but its raw value is needed every time a webhook is signed, like a provider key. It is therefore encrypted with AES-256-GCM at rest, decrypted just in time for each signing operation, and kept only in memory while in use.

Surviving unreliable providers with ordered fallback

Any system tied to a single provider inherits that provider's outages, throttling and timeouts. The service reduces that exposure with a per-customer priority list. A customer can register several providers, for example SendGrid, Resend and Postmark, and rank them. When a send request does not name a provider, the attempt list is built from that ranking, highest priority first.

The send walks down the list. If a provider fails because it is down, throttling or too slow, the failure is logged and the next provider is tried. The first success ends the walk, and the customer never notices the fallback unless every provider fails.

Customers can also pin a message to one provider, optionally with a single named fallback; in that case only that pair is attempted. Not every error is treated equally:

• Permanent failures, such as a rejected credential or a malformed request, fail immediately, because retrying cannot fix them.
• Transient failures, such as timeouts, rate limits and provider 5xx responses, are retried with backoff.

When the entire list fails, the error bubbles up to the queue, whose own retry policy takes over. Fallback does not make provider failures disappear; it gives each message more chances to succeed before a failure becomes visible to the customer.

One subtlety worth planning for: a timeout does not prove the provider rejected the message. A provider may accept a request and then fail to answer in time, and falling back at that point can deliver the same notification twice. Passing an idempotency key where the provider supports one, or accepting occasional duplicates for low-stakes messages, are the usual ways to handle this. Our article on idempotency keys for Node.js POST endpoints covers the pattern in depth.

Queue architecture

Delivering a notification inside the API request that triggered it ties the caller's latency to the slowest provider. A sluggish or unavailable provider would keep the customer's HTTP call hanging until it eventually succeeds or times out. The service avoids this by splitting acceptance from delivery.

Decoupling the request from the delivery

The API only validates the request, creates a Job row in the database and enqueues the work. BullMQ on Redis does the scheduling, with a queue per kind of work:

• notifications-email for customer emails
• notifications-webhook for customer webhooks
• platform-email for the service's own internal alerts
• notifications-failed as the dead-letter queue for jobs that have run out of retries

Separate queues mean one type of work can back up or be tuned without affecting the others. For a broader treatment of BullMQ job design, see building reliable background job systems with BullMQ and Redis.

Retries count whole passes, not providers

Each job gets three attempts with exponential backoff, and the important detail is what an "attempt" means. It is not one try per provider. A single attempt is a full pass over the customer's providers in priority order, stopping at the first success. When every provider in a pass fails, the job has used up one attempt, and the following retry begins once more with the highest-priority provider. Only after three failed passes is the job copied, error included, into the dead-letter queue, and the original marked FAILED.

The database is the source of truth even when the queue itself breaks. A customer Job, or a PlatformEmailLog for internal mail, is written with status PENDING before anything is enqueued. If Redis is unreachable or adding the job throws, the error is logged, the failure is reported back, and the row is updated to FAILED with the reason. No request disappears silently between the API and the queue.

What the dead-letter queue is for

When a job exhausts its three attempts, a copy lands in notifications-failed containing the full payload, the error and a timestamp, while the database record is set to FAILED with the same reason. Recording the failure twice is intentional, because each record serves a different audience:

• The database row is customer-facing and backs GET /jobs/:id.
• The DLQ entry is operator-facing, a disposable copy intended to be monitored and acted on.

A DLQ is only valuable if something watches it. Every job that arrives triggers a Slack alert, reusing the alerting path already built for platform email failures. That turns the queue into an early-warning signal: a burst of dead-lettered jobs in a short window usually points to something systemic, such as a provider outage or a bad deploy.

Nothing leaves the DLQ automatically. A dead job is retried only when the customer calls POST /jobs/:id/retry, which resets it to PENDING and restarts the flow.

Priority and concurrency

Jobs carry one of three priorities, CRITICAL, NORMAL or LOW, mapped onto BullMQ's native priority support. Free-plan traffic is always forced to LOW so it cannot compete with paying traffic. Platform emails skip that logic entirely: they always enqueue as CRITICAL and use a shorter backoff.

Worker pools pull jobs continuously, each running with a concurrency of five, so up to five jobs per pool are processed at the same time.

Ordering: a deliberate gap

The service does not guarantee per-customer ordering. Concurrency and priority are tuned for throughput, and with five parallel workers plus retries, two messages sent in sequence can easily be delivered in reverse. For most notifications this is harmless because they are independent. It becomes a problem only for causally linked pairs where the sequence carries meaning, such as a message that should logically follow another one.

There are two general remedies, each with a price:

• Serialize per customer. Route a customer's jobs through a single lane so order is guaranteed, at the cost of that customer's parallelism.
• Make messages order-tolerant. Attach timestamps and let consumers interpret sequence themselves, which shifts responsibility to how customers read deliveries.

The limitation is known and was accepted in favor of throughput. Current usage does not justify either cost yet, so the decision is deferred until real customer data shows which trade-off fits better. Documenting a gap like this explicitly is often better than half-solving it.

Deliverability: preventing failures before and after the send

Emails can bounce, land in spam or vanish depending on sender reputation and signals that accumulate over time. The service addresses this at two points.

Before the send: domain verification

When a customer wants to send from their own domain, the service registers that domain with the customer's chosen provider and receives the DNS records that prove ownership. These records differ by provider, so they are shown to the customer to add at their DNS host. Sending from that domain stays blocked until verification passes.

After the send: acting on provider events

Providers report deliveries, opens, clicks, bounces and spam complaints via webhooks. The service stores these events and responds according to type:

• A hard bounce (the address does not exist or is permanently refused) suppresses the recipient automatically, and later jobs to that address are blocked before any provider is called.
• A spam complaint triggers the same suppression.
• A soft bounce, such as a full mailbox or a temporarily unreachable server, is only logged, because the problem is likely temporary.

The rule is that only signals indicating a permanent problem lead to a block. This does not solve deliverability outright, since reputation grows and decays based on factors beyond any single message, but it stops the system from repeatedly hitting addresses that are known to fail, which is itself one of the factors that damages reputation.

Keeping tenants from hurting each other

All customers share the same queues, worker pools and Redis instance; nothing is physically isolated. Fairness comes from three controls applied at different stages.

Gate one: per-customer rate limits at the API

Before a request becomes a job, it passes a Redis-backed rate limit per customer over a 60-second window, sized by plan:

• FREE: 5 requests per minute
• INDIE: 50 requests per minute
• STARTUP: 200 requests per minute

A sudden spike from one account is rejected at the edge and never floods the queue.

Gate two: priority once accepted

Accepted jobs are processed by priority, and free-plan jobs are always LOW. Workers always take higher-priority jobs first, so even if a free account builds a backlog, paid customers' notifications move ahead of it.

Gate three: monthly quotas

Over a longer horizon, quotas cap volume. The free plan allows 100 combined email and webhook sends per month. Paid plans cap only webhooks; email is unlimited for them.

What is still missing

There is no hard resource isolation: no dedicated workers per customer. Two customers on the same priority tier share the same capacity, and within a tier jobs are handled roughly first-come, first-served. At present volume this has not mattered, as traffic sits comfortably within what the shared infrastructure handles and no tenant has noticeably slowed another. At larger scale, per-tenant isolation or a fair scheduling scheme (for instance round-robin across tenants within a priority level) would become necessary.

Verifying webhooks in both directions

Every integrated provider pushes events back via webhooks: delivery confirmations, bounces, spam reports and payment status changes. Each receiving endpoint is a public URL. Without verifying the sender, anyone could post a fake "delivered" event, forge a bounce to get someone else's address suppressed, or, most damaging, fake a payment event to mark a subscription as paid.

Inbound: one scheme per provider

There is no common standard, so each integration is its own piece of security work:

• SendGrid signs payloads with ECDSA, and the signature is checked against SendGrid's public key before the payload is trusted.
• Resend delivers webhooks through Svix, which uses a timestamped HMAC scheme that protects against both tampering and replay.
• Postmark does not sign webhooks, so its endpoint relies on HTTP Basic Auth with a per-customer secret.
• Paystack payment webhooks are verified with HMAC using SHA-512.
• Polar payment webhooks are validated using the standardwebhooks library.

The Postmark case carries an easy-to-miss trap: the secret must be compared in constant time, for example with Node's crypto.timingSafeEqual, not with a normal string equality check. An ordinary comparison can return as soon as a character differs, and those timing differences can let an attacker recover a secret one character at a time instead of brute-forcing the whole value.

None of these schemes are interchangeable, and each integration has to be learned and implemented from scratch. The invariant that holds across all of them is that nothing arriving at a webhook endpoint is acted upon until its origin is verified.

Outbound: letting customers trust the platform

The same problem exists in reverse when the service delivers webhooks to customer endpoints. If a customer has configured a signing secret, each delivery includes an X-Webhook-Timestamp header and an X-Webhook-Signature header. The signature is an HMAC-SHA256 computed over the string timestamp.payload, which binds the body to a specific moment. A receiver recomputes the signature and also rejects timestamps that are too old, which is what stops a captured request from being replayed later.

Customers fully control their secret: they can generate it, check its status, rotate it or delete it. Without a configured secret, webhooks are still delivered, just unsigned. That favors backward compatibility, but it also means receivers must opt in to verification to be protected. For a complementary view on the receiving side, see outbox and inbox tables for webhooks that survive failure.

Key takeaways

• Decide between encryption and hashing by asking whether the original value must ever be recovered.
• Treat provider fallback and queue retries as separate layers, and define precisely what one "attempt" means.
• Write a durable database record before enqueueing so a queue outage cannot lose a request.
• A dead-letter queue earns its keep only when alerts fire on it and a clear retry path exists.
• Suppress only on permanent signals such as hard bounces and spam complaints.
• Layer fairness controls (rate limits, priority, quotas) and be explicit about what isolation you do not yet provide.
• Verify every inbound webhook with the provider's own scheme, use constant-time comparisons, and sign outbound webhooks with a timestamp to prevent replay.
• Name your known limitations, such as ordering, and defer expensive fixes until data shows which trade-off fits.