Designing a Duplicate-Free Notification Pipeline on SQS, SES and Redis

Two workers receive the same event at the same moment. Each checks the email log, finds no record, and sends. A supplier ends up with the same purchase order twice, and the team learns about it from a support ticket rather than from monitoring. This article walks through a notification engine for a B2B supply-chain platform that grew to handle more than five million events a month, and the three production failures that shaped it: lost emails, emails blocked on PDF generation, and duplicates.

What the engine has to deliver

In a food supply-chain SaaS, business events such as creating or editing an order, or running an inventory operation, must produce documents for the right people: purchase orders, sales orders, delivery orders, goods receipt notes and invoices. A single event can therefore fan out into several steps: render a PDF, store it, and email it to multiple recipients. The same engine also sends SMS and WhatsApp messages for a handful of other use cases.

The stakes are concrete. A notification that never arrives is not a cosmetic bug; a missing purchase order can hold up real stock on its way to a customer.

Two queues: one decides, one delivers

The design rests on two Amazon SQS queues with clearly separated jobs.

The primary queue is where decisions happen. A single consumer reads every incoming event and works out what it implies: which document to produce, which channel to use and who should receive it.

The sub-queue only delivers. Delivery workers apply no business rules; they move messages to providers.

One message per recipient

The most important decision in this layout is that every recipient gets their own message on the sub-queue. An order that must reach ten people becomes ten independent messages. If delivery to one address fails, the other nine still go out, and a retry targets only the failed recipient instead of resending to everyone.

This split also isolates change. Swapping in a new provider or channel touches only the delivery side; the code that decides what to send never needs to know how a message is transported.

Why not one queue per channel?

A separate queue for email, SMS and WhatsApp would give each channel independent throughput and failure handling. At this volume it was not necessary, and adding a queue when the numbers call for it is cheaper than operating four from the start.

Failure one: emails that never arrived

Email providers fail, sometimes loudly with an error and sometimes quietly with a response that lacks the expected fields. Amazon SES is the primary provider here, and a response without a message ID is treated as a failure just like an exception. In either case, the same email is sent through SendGrid instead:

// simplified
try {
  const res = await ses.send(email);
  if (!res.MessageId) throw new Error("No MessageId from SES");
} catch (err) {
  await sendgrid.send(email); // fallback
}

Notice the explicit throw when MessageId is missing: it routes the ambiguous case into the same fallback path as a hard error. The trade-off is worth understanding. If SES accepted the email but the response was lost, the fallback can deliver it a second time, so this layer favours "delivered at least once" over "delivered at most once". For business documents that is usually the right bias, but it is another reason the deduplication step described below matters.

When both providers fail, the message goes to a dead-letter queue and is retried after an interval. On top of that, a separate resend mechanism picks up emails that are recorded as missed, so there is a path back for anything that slipped through.

Failure two: PDFs that failed under load

Document emails follow a strict rule: nothing is sent until the PDF exists. Recipients need the actual purchase order, not a link that returns an error. The consumer invokes a PDF-generation Lambda, writes the file to S3 and only then sends the email containing a link to it.

The rule is correct, but it created a hidden dependency. When the system lagged and PDF generation failed, the email did not fail visibly; it simply never happened, because it was waiting for a file that would never be produced. The fix was to give PDF generation its own retries, so a transient failure in rendering no longer silently cancels the downstream notification.

Failure three: the same email, twice

Duplicates came from three distinct sources, and each needed a different response:

• ordinary logic bugs, which were fixed as bugs;
• retries firing for messages that had in fact already been delivered;
• the race between two consumers handling the same event concurrently.

Why a log check is not enough

From the beginning the pipeline consulted the email log before sending: if a record showed the message had gone out, the retry was skipped. That handles sequential retries well. It cannot stop the race, because reading the log and writing to it are two separate operations. Two workers can both read "not sent" before either one writes, and both proceed. SQS standard queues deliver at least once, so the same message arriving at two consumers is expected behavior, not a rare accident.

An atomic lock in Redis

The fix was to put an atomic lock in Redis in front of the log check. The key combines the order ID, the recipient's email address and the document type, and it expires after roughly five minutes:

// simplified
const key = `notif:${orderId}:${email}:${docType}`;
const locked = await redis.set(key, "1", "NX", "EX", 300);
if (!locked) return; // someone else already has it

SET with the NX option writes the key only if it does not already exist, and EX 300 sets a 300-second expiry in the same command. Because the existence check and the write happen as a single operation on the Redis server, exactly one worker gets a truthy result; every other worker sees null and returns early. That closes the gap the log check left open.

A few properties of this approach are worth keeping in mind:

• The key granularity matters. Including the document type means a purchase order and an invoice for the same order and recipient are not mistaken for duplicates.
• The TTL is a safety valve. If a worker crashes after taking the lock, the key expires and a later retry can proceed. The expiry should comfortably exceed the time a send normally takes, and ideally the SQS visibility timeout, or a slow worker's lock can lapse while it is still sending.
• The lock covers concurrent duplicates; the persistent log still catches later ones.

The same idea appears in HTTP APIs as idempotency keys; the article on idempotency keys in Node.js POST endpoints covers that side.

Where the system ended up

After these changes, the engine processes more than five million events a month across order, inventory and document triggers, with 98.9% successful delivery and a p95 latency under 1.2 seconds.

What to do differently next time

• Detect failures before customers do. Too many problems surfaced first as complaints and only afterwards in logs. Alerts on delivery failures and on dead-letter queue depth belong in the first release, not a later one.
• Stop events from disappearing. Some events went missing for reasons that were never fully explained. A transactional outbox, which records the event alongside the business write and relays it afterwards, makes lost events much rarer. The article on outbox and inbox tables explains the mechanics.
• Keep the single sub-queue until it hurts. The shared delivery queue has held up; it only needs splitting if one channel begins to starve the rest.

Checklist for building your own

• Keep routing decisions in one place and delivery in another.
• Emit one message per recipient so a single failure cannot block the rest.
• Do not trust a single provider, and treat a missing message ID as a failure.
• Give failed messages a destination: a dead-letter queue with retries.
• Never send a document email before the document exists, and retry the step that produces it.
• Make deduplication atomic, because check-then-write leaves a window.
• Alert on failures so customers are not your monitoring system.

Wrapping up

None of these protections was designed upfront; each was added after something broke in production, which is how most delivery pipelines actually mature. The lesson is less about any single fix and more about the order of thinking: assume at-least-once delivery, assume providers and dependencies will fail quietly, and make every step either idempotent or guarded by something atomic.