Outbox and Inbox Tables: Designing Webhooks That Survive Failure

Webhooks look like the simplest integration there is: one side makes an HTTP POST, the other side handles it. In practice they carry every hazard of a distributed system, because the network between two services can drop requests, time out halfway, deliver the same payload twice, or reorder events. Treat a webhook like an ordinary CRUD call and you will eventually lose notifications, fire side effects twice, and end up with two systems that disagree about what happened.

This guide walks through a design that holds up under those conditions. You will see why the naive approach breaks, how a transactional outbox makes outgoing webhooks reliable, how an idempotent inbox makes incoming webhooks safe to retry, how to cope with events that arrive in the wrong order, how to quarantine payloads that can never succeed, and which managed services on AWS, Azure and Google Cloud fit each piece.

Why the obvious implementation loses data

Consider a SaaS backend handling a meaningful state change, such as an order being fulfilled or a subscription becoming active. A partner calls your API to confirm the action, and your service now has two jobs:

1. Persist the new state, for example setting the entity status to Active.
2. Tell a downstream service that the entity is ready, by sending it a webhook.

The intuitive code writes to the database and then, on the next line, fires the HTTP request. That is a dual write: two independent systems are updated one after the other, with nothing tying them together.

Two failure modes follow directly from that:

• The process dies between the two steps. The database now says the entity is active, but the request was never sent. Your records are correct, the downstream service knows nothing, and nobody notices until a customer complains.
• The request goes out, then the transaction fails. The downstream service has been told the entity is active, but your database rolled back and still considers the operation failed.

Neither ordering fixes the problem. Put the HTTP call first and you get the second failure; put it last and you get the first. The root cause is that a database commit and a network call cannot be made atomic together, so any crash or error in between leaves the two sides out of sync.

Sending webhooks reliably with a transactional outbox

The outbox pattern removes the dual write by never making the HTTP call from the request path at all. Instead, the intention to send the webhook becomes data, and that data is written in the same database transaction as the business change. Either both are committed or neither is.

The four steps of the outbox flow

1. Open a transaction. The business operation starts a normal database transaction.
2. Write the state and the event together. Inside that transaction the service updates the entities table (for instance, setting the status to Active) and inserts a row into outbox_events holding the exact payload the downstream service should receive.
3. Commit. Once the transaction commits, the database durably records both the new state and the promise to announce it.
4. Relay the event. A separate background worker, usually called the relay or publisher, repeatedly looks for outbox rows that have not been sent yet. For each one it performs the HTTP POST and then marks the row as processed.

The outbox table

The table below stores one row per pending notification. aggregate_type and aggregate_id identify which business object the event is about, event_type names what happened, payload holds the body to deliver, and processed_at stays empty until the relay confirms delivery. Querying for rows where processed_at is null gives the relay its work list. Note that the inline comments use a single dash; in PostgreSQL a comment needs two (--), so fix that before running the statement.

CREATE TABLE outbox_events (
id UUID PRIMARY KEY,
aggregate_type VARCHAR(50), - e.g., 'Order' or 'User'
aggregate_id UUID, - e.g., Entity ID
event_type VARCHAR(100), - e.g., 'order.activated'
payload JSONB NOT NULL, - The exact webhook payload
created_at TIMESTAMP DEFAULT NOW(),
processed_at TIMESTAMP - Null until successfully sent
);

What the outbox guarantees, and what it does not

If the server crashes after the commit, nothing is lost: the row is still in the table and the relay will find it on its next pass. If the downstream endpoint is unavailable, the relay simply retries, ideally with exponential backoff so a struggling receiver is not hammered. The state change and the intent to notify can no longer diverge.

The trade-off is that delivery becomes at least once. A relay can send the request successfully and then crash before it updates processed_at, in which case the same event goes out again on the next run. That is acceptable only if receivers deduplicate, which is exactly what the inbox pattern on the other side provides. Including the outbox row's id in the payload or a header gives receivers a stable key to deduplicate on. If you run several relay instances, make sure two workers cannot claim the same row at the same time; in PostgreSQL, selecting rows with FOR UPDATE SKIP LOCKED is a common way to do that.

Receiving webhooks safely with an idempotent inbox

Now flip the perspective to webhooks your service receives from partners or upstream systems.

Suppose your handler takes five seconds because it performs heavy computation or waits on a lock held by another service. The sender's HTTP client may give up before you respond, conclude that you never got the event, and send it again. Now the same event arrives twice. If your handler sends an email or creates a record every time it runs, the customer gets two emails and you get a duplicate row.

The inbox pattern separates accepting a webhook from acting on it.

The four steps of the inbox flow

1. Receive and verify. As soon as the request arrives, check its HMAC signature so you know it really came from the partner and was not forged or altered.
2. Store the raw payload. Insert the untouched JSON body into a webhook_inbox table, keyed by the partner's own unique event identifier and protected by a database uniqueness constraint.
3. Acknowledge right away. Return 200 OK immediately, before any business logic runs.
4. Process in the background. A worker picks up pending inbox rows, checks whether each event has already been handled, skips it if so, and otherwise runs the business logic and marks the row as processed.

The inbox table

Here each row records who sent the event (partner_name), the sender's identifier for it (partner_event_id), the payload, whether the signature checked out, and a status that moves through PENDING, PROCESSED or QUARANTINED. The important line is the composite UNIQUE(partner_name, partner_event_id) constraint: it is what turns duplicates into harmless no-ops. As with the outbox table, the single-dash comments need to become -- for PostgreSQL to accept the statement.

CREATE TABLE webhook_inbox (
  id UUID PRIMARY KEY,
  partner_name VARCHAR(50), - e.g., 'Stripe' or 'GitHub'
  partner_event_id VARCHAR(100), - The unique ID from the sender
  payload JSONB NOT NULL,
  signature_verified BOOLEAN,
  status VARCHAR(20), - 'PENDING', 'PROCESSED', 'QUARANTINED'
  received_at TIMESTAMP DEFAULT NOW(),
  processed_at TIMESTAMP,
  UNIQUE(partner_name, partner_event_id) - Prevents duplicate inserts
);

Why the constraint does the heavy lifting

Because the handler only verifies, inserts and returns, it responds quickly and the sender rarely times out in the first place. When a sender does retry, even ten times in a row, the uniqueness constraint lets exactly one insert succeed. Your handler should treat the resulting unique-violation error (or an ON CONFLICT DO NOTHING result) as success and still return 200 OK, otherwise the sender will keep retrying an event you already have. Since only one row exists, the worker runs the side effects only once.

Two details are worth getting right. First, deduplication depends on the partner supplying a stable event identifier; most webhook providers include one, but confirm this for each integration. Second, a worker can crash after performing the side effect but before marking the row processed, so wherever possible run the business change and the status update in one transaction, and make external side effects idempotent too. For a deeper look at deduplicating requests with keys, see idempotency keys in Node.js POST endpoints.

Handling events that arrive out of order

Even with duplicates under control, there is no guarantee that events arrive in the order they were produced. Your service might receive entity.completed before entity.started. A handler that blindly applies each event will then try to move an entity from draft straight to completed, which either corrupts its state or fails with something like a 409 Conflict.

Checking each transition against the state machine

The fix is to stop treating events as commands to mutate state and start treating them as proposed transitions that must be validated. This is sometimes described as a state reconciliation engine, in the spirit of event sourcing: the worker compares the incoming event with the entity's current state and decides whether the transition is legal.

The sketch below shows that decision. If a completion event arrives while the entity is still a draft, the prerequisite has not happened yet, so the function reports the event as deferred instead of applying it. The comments name two ways to handle a deferral: leave the row in the inbox and retry it later, or record a projected state and wait for the missing event. A start event on a draft entity is a valid transition and is applied. Treat this as pseudocode: return status: 'DEFERRED'; is not valid JavaScript and should be return { status: 'DEFERRED' };, and a real implementation would also handle the remaining combinations of event and state.

function processWebhookEvent(event, currentEntityState) {
    if (event.type === 'entity.completed' && currentEntityState === 'draft') {
        // The 'started' event hasn't arrived yet!
        // We cannot transition from 'draft' directly to 'completed'.

        // Option A: Leave it in the inbox and retry in 5 minutes.
        // Option B: Store a "Projected State" and wait for the missing piece.
        return status: 'DEFERRED';
    }

    if (event.type === 'entity.started' && currentEntityState === 'draft') {
        return transitionTo('started');
    }
}

Deferral as a self-healing loop

Take an order where the "shipped" event reaches you before the "paid" event. Applying "shipped" immediately would put the order into a state your model does not allow. With a state-aware processor the sequence becomes:

1. "Shipped" arrives, the evaluator sees that payment is missing, and the event is deferred.
2. "Paid" arrives, is valid, and updates the order.
3. The deferred "shipped" event is retried, now finds its prerequisite satisfied, and is applied.

Deferred events can sit in a dedicated retry queue, for example Amazon SQS or a Redis-backed queue, and a background worker periodically retries them. The result is a workflow that refuses invalid transitions yet eventually converges on the correct state without dropping any event. Put a limit on how long an event may stay deferred, though: if the prerequisite never arrives, the event should eventually be treated as a failure rather than retried forever, which leads to the next section.

Isolating poison pills with retries and a dead letter queue

Some events will never succeed no matter how often you retry them: a malformed payload, or a reference to an ID that does not exist in your database. These are called poison pills. A naive worker retries them forever, and if the queue is processed in order, one bad message can block every valid event behind it.

The standard defence is a bounded retry policy with increasing delays, followed by a dead letter queue (DLQ). A typical schedule looks like this:

1. First attempt fails; wait one minute.
2. Second attempt fails; wait five minutes.
3. Third attempt fails; wait fifteen minutes.
4. Fourth attempt fails; move the event to the DLQ.

The DLQ can be a table in your own database or a feature of a managed queue. What matters is what happens next: events in the DLQ should appear on an internal admin view and raise a high-priority alert, because each one represents data your system could not handle. An engineer investigates, fixes the mapping bug or the bad data, and then replays the event so it flows through the normal processing path. Build that replay action early; without it, recovering from a DLQ becomes a manual database edit under pressure.

Mapping the design onto AWS, Azure and Google Cloud

The outbox and inbox live in your relational database, but the surrounding machinery (ingress, queues, workers, DLQs) maps well onto managed cloud services, which removes much of the operational burden. The shape is the same on each provider; only the product names change.

AWS

• Ingress: Amazon API Gateway accepts incoming webhooks, with a Lambda authorizer checking the HMAC signature before the request reaches the backend.
• Database: Amazon Aurora PostgreSQL holds the business tables together with webhook_inbox and outbox_events, so the transactional guarantees apply.
• Queues and DLQ: An SQS standard queue drives asynchronous processing, and a configured SQS dead letter queue receives messages once they exceed the maximum receive count. Standard queues are themselves at-least-once and do not preserve order, which is one more reason the idempotency and state checks above matter.
• Workers: Lambda functions triggered by SQS process inbound events. The outbox relay runs as a scheduled Lambda or an ECS Fargate task that polls Aurora every few seconds, sends pending events and sets processed_at.

Azure

• Ingress: Azure API Management receives webhooks, validates signatures and forwards requests to the backend.
• Database: Azure Database for PostgreSQL Flexible Server stores application state plus the inbox and outbox tables.
• Queues and DLQ: Azure Service Bus brokers the messages and has built-in dead-lettering, moving a message aside automatically after a configured number of delivery attempts.
• Workers: Azure Functions with Service Bus triggers process inbox payloads. The outbox relay runs as a background loop in Azure Container Apps, or as a Kubernetes CronJob if you are on AKS, polling PostgreSQL for unsent events and delivering them over HTTP.

Google Cloud

• Ingress: Google Cloud API Gateway handles incoming HTTP webhooks and authentication.
• Database: Cloud SQL for PostgreSQL stores the relational data, including both tables.
• Queues and DLQ: Pub/Sub routes messages asynchronously. The main subscription processes events, and a dead letter topic captures messages that are still unacknowledged after the configured maximum number of delivery attempts.
• Workers: Cloud Run services, able to shrink to zero instances between bursts, receive Pub/Sub push deliveries to process the inbox. The outbox relay is either a Cloud Run job or a Cloud Run service invoked on a schedule by Cloud Scheduler, polling Cloud SQL and sending pending events.

For more patterns around connecting services, such as OAuth and resilient API calls, see six integration patterns for connecting Node.js services.

Key takeaways

• A reliable webhook system is an event-processing pipeline, not a pair of HTTP endpoints.
• Never update the database and call a remote service as two unrelated steps; write an outbox row in the same transaction and let a relay deliver it.
• The outbox gives at-least-once delivery, so every receiver must deduplicate.
• On the receiving side, verify, store with a uniqueness constraint on the sender's event ID, acknowledge immediately, and do the real work in a worker.
• Validate each event against your state machine and defer those whose prerequisites are missing, with a limit on how long they wait.
• Cap retries, route persistent failures to a DLQ with alerting, and make replay a first-class operation.
• Managed queues such as SQS, Service Bus and Pub/Sub supply retries and dead-lettering, while the database tables keep the guarantees that matter.