Webhooks & Events

Sending an email is only the first half of the story. This chapter answers what happens after the message leaves your provider, how you find out about it, and what to do with each event. Webhooks are how your application learns in real time whether an email was delivered, bounced, opened, clicked, or marked as spam. Instead of you polling the provider asking "what happened to message X?", the provider calls you the moment something happens.

Quick answers to the questions this chapter covers:

• What is a webhook, and why do I need one?
• Which event types will I receive, and which ones must I act on?
• How do I build a webhook endpoint that does not get me into trouble?
• How do I verify signatures so nobody can forge events?
• How do I handle bounces, complaints, and deliveries?
• Why must my processing be idempotent?
• What happens when my endpoint fails, and how do retries work?
• How do I test webhooks before going live?

This chapter is deliberately provider-neutral. Every example uses either raw standards (Node's built-in crypto, the Standard Webhooks signature scheme, plain HTTP) or a thin provider / emailClient abstraction you own. Provider names, Amazon SES, Postmark, SendGrid, Mailgun, Resend, Brevo, SparkPost, Mandrill, appear only as concrete examples of how a particular vendor spells a thing. Nothing here assumes a specific ESP, and the patterns transfer unchanged when you switch vendors.

What is a webhook, and why does my email system need one?

A webhook is an HTTP endpoint you own that your email provider calls (with a POST request) whenever something happens to a message. It is the reverse of an API call: normally your app calls the provider, but with webhooks the provider calls your app. That push model is the only practical way to learn about events that happen on someone else's mail server, in someone else's mailbox, minutes or hours after you sent the message.

You need webhooks for three concrete reasons:

• Your suppression list stays accurate. Bounces and complaints arrive as events, and you use them to stop mailing addresses that will hurt you.
• You catch deliverability problems early, before a rising bounce or complaint rate sinks your sender reputation.
• You can measure engagement (opens, clicks) and show real delivery status in your product.

An app that sends email but ignores webhook events is flying blind. It keeps mailing addresses that bounce and recipients who already complained, which is exactly how a sending domain gets blocklisted. This chapter closes the loop on the reliability work from the previous chapter. In Sending Reliability you made sure every email is sent exactly once and you stored each provider message ID. Webhooks are where those IDs pay off: each incoming event references a message, letting you update its status, prune your lists, and react to problems automatically.

Why not just poll the provider's API?

Polling is the obvious alternative, and it is the wrong default. To learn about delivery status by polling, you would have to ask the provider "what happened to message X?" repeatedly for every message in flight, because you cannot know in advance when (or whether) a bounce will arrive. The economics do not work:

• A soft bounce or a complaint can arrive seconds after the send, or hours later (some mailbox providers batch complaint feedback). To catch the slow ones you must keep polling each message for hours, which multiplies your request volume by the polling frequency.
• Most providers rate-limit their status APIs and many do not retain per-message status for long. By the time you poll, the event may already have aged out.
• Polling is pull-based and lossy by design: if you stop polling a message too early, you simply never learn what happened to it.

Webhooks invert all of this. The provider tells you exactly once, the moment the event is known, and retries until you acknowledge. Reserve polling for reconciliation, a periodic sweep that catches events a webhook outage may have dropped (covered under failure modes below), not as the primary mechanism.

Webhooks versus inbound feedback channels

Provider webhooks are not the only way delivery signals reach you. Two other channels exist and are worth understanding because they sometimes feed the same suppression logic:

• DSNs (Delivery Status Notifications, RFC 3464). When you send over raw SMTP without a managed ESP, hard and soft bounces come back as machine-readable bounce messages to your Return-Path (envelope MAIL FROM) address. Your provider normally parses these for you and re-emits them as bounced webhook events; if you run your own MTA you parse the DSN yourself.
• Feedback loops (FBLs / ARF, RFC 5965). Several mailbox providers send a copy of spam complaints back to the sender in the Abuse Reporting Format. Your ESP enrolls in these on your behalf and surfaces them as complained events. If you send directly, you enroll yourself (for example via the relevant provider's postmaster program) and parse ARF reports into the same complaint-handling path described later in this chapter.

The point: a bounced or complained webhook is usually your provider's normalized view of an underlying SMTP DSN or an ARF feedback report. The handling logic in this chapter is identical regardless of which channel produced the signal.

The examples below follow the Standard Webhooks signature scheme, an open, vendor-neutral spec for signing webhook payloads, but the architecture is universal. Every major provider sends signed JSON over HTTPS POST and retries on failure. Event names and signature headers differ between providers; the patterns do not.

Which event types will I receive?

Each event tells you something different about an email's journey. Treat them as two groups: health signals (bounced, complained) that demand action to protect your reputation, and observability signals (sent, delivered, opened, clicked) that feed status tracking and analytics.

The names above are illustrative. The exact strings vary by provider, and mapping them onto a small internal vocabulary is the first job of your webhook layer.

Provider event-name mapping

Normalize every provider's vocabulary into your own canonical set the moment a payload arrives, so the rest of your code never branches on vendor-specific strings. The table below shows how several common providers spell each category. Treat it as a starting point, not gospel, verify against your provider's current docs, because vendors add and rename events.

A small normalization function keeps the vendor strings contained to one place:

// Map a provider-specific event payload onto your canonical vocabulary.
// Keep this the ONLY place that knows vendor-specific strings.
type CanonicalType =
  | 'sent' | 'delivered' | 'bounced' | 'complained' | 'opened' | 'clicked' | 'unsubscribed';

interface CanonicalEvent {
  id: string;            // stable, provider-supplied event id (for idempotency)
  type: CanonicalType;
  messageId: string;     // your stored provider message id
  email: string;
  bounceClass?: 'hard' | 'soft';
  occurredAt: Date;
  raw: unknown;          // keep the original payload verbatim
}

function normalize(providerName: string, payload: any): CanonicalEvent {
  switch (providerName) {
    case 'ses': {
      const m = payload.mail;
      const typeMap: Record<string, CanonicalType> = {
        Delivery: 'delivered', Bounce: 'bounced', Complaint: 'complained',
        Open: 'opened', Click: 'clicked', Send: 'sent',
      };
      return {
        id: payload.mail.messageId + ':' + payload.notificationType,
        type: typeMap[payload.notificationType],
        messageId: m.messageId,
        email: (m.commonHeaders?.to ?? [''])[0],
        bounceClass: payload.bounce?.bounceType === 'Permanent' ? 'hard' : 'soft',
        occurredAt: new Date(m.timestamp),
        raw: payload,
      };
    }
    // case 'sendgrid': ...  case 'postmark': ...  case 'resend': ...
    default:
      throw new Error(`No normalizer for provider ${providerName}`);
  }
}

Which events can I safely ignore, and which can I not?

bounced and complained are the events you cannot ignore. They directly affect deliverability and, for complaints, legal and ethical obligations. The other four are valuable but optional: you can ship a working system that only handles bounces and complaints, then add the rest for analytics later. If you only ever wire up two handlers, make them these two.

There is a third event worth promoting to "must handle" if your provider emits it: an unsubscribe event tied to one-click list-unsubscribe. Under RFC 8058 (List-Unsubscribe-Post), a recipient can unsubscribe from a bulk message in one click directly from the mailbox UI, and the mailbox provider POSTs to your List-Unsubscribe URL. Many ESPs that host that URL on your behalf surface the result as a webhook event. If you receive it, treat it like a complaint-lite: suppress the address for that mail stream immediately. See Compliance for the legal framing and Marketing Emails for where one-click unsubscribe is mandatory.

What is the difference between sent and delivered?

sent means the provider accepted your request and queued the message. delivered means the recipient's mail server accepted it. The gap between them is exactly where bounces happen. Do not tell a user "delivered" when you have only seen sent. A message can sit in sent for seconds or minutes and still bounce.

It helps to think of the lifecycle as a small state machine, because some transitions are terminal and some are not:

queued ──► sent ──► delivered ──► opened ──► clicked
                │         │
                ├──► bounced (hard)  [terminal, suppress]
                ├──► bounced (soft)  [retryable, count]
                └──► complained      [terminal, suppress]

delivered is not terminal: a complaint almost always arrives after delivery, because a human has to receive the message before they can mark it as spam. So a single message legitimately produces sent → delivered → complained. Your status model must allow a delivered message to later become complained; do not freeze status at delivered.

Does delivered mean the recipient read the email?

No. Delivery means the message reached the recipient's server, not that a human saw it. Only opened hints at that, and even opened is approximate: it relies on a tracking pixel, which privacy features and image blocking can suppress. Apple Mail Privacy Protection, for example, pre-fetches images and can register an "open" the recipient never made. Treat open rates as a directional signal, never as proof a specific person read a specific email.

clicked is a stronger signal than opened (a machine pre-fetching images rarely follows links), but it has its own caveats: link-tracking redirects can be triggered by security scanners that "click" every link in an inbound message to check for malware. If you depend on engagement for anything consequential, sunset policies, re-engagement campaigns (see List Management), corroborate opens with clicks and replies rather than trusting opens alone.

How do I set up a webhook endpoint correctly?

There are three steps: create the endpoint, verify signatures on every request, and register the URL with your provider. Get all three right and the rest of the chapter (processing, idempotency, retries) builds on top.

Step 1: How should the endpoint itself behave?

Your endpoint must:

• Accept POST requests.
• Return a 2xx status quickly (within 5 seconds).
• Handle duplicate events (idempotent processing, covered below).

app.post('/webhooks/email', async (req, res) => {
  // Return 200 immediately to acknowledge receipt
  res.status(200).send('OK');

  // Process asynchronously
  processWebhookAsync(req.body).catch(console.error);
});

The single most important pattern here is acknowledge first, process later. The provider gives you a tight budget, return 2xx within about 5 seconds, and it interprets a slow or failed response as "delivery failed", which triggers retries (more on that below). If your handler does its real work (database writes, list updates, sending follow-up emails) before responding, a slow database or a downstream hiccup turns into a timeout, a retry, and ultimately duplicate processing.

So the handler does two things in order: (1) send 200 immediately to acknowledge receipt, then (2) kick off the actual processing asynchronously. Note the .catch(console.error). Because the response has already been sent, an error in async processing cannot be reported back to the provider, so you must catch and log it yourself or the failure vanishes silently. In a real system, "process asynchronously" should mean enqueue to a durable job queue, not just fire a background promise. See the Sending Reliability chapter for the durability argument.

The crucial ordering bug: verify before you acknowledge

The snippet above is simplified for the acknowledge-first idea, but it hides a trap. You must verify the signature before you return 200, and you must capture the raw body before any JSON body parser touches it. The correct order inside the handler is:

1. Read the raw request bytes.
2. Verify the signature against those bytes (reject with 4xx if invalid).
3. Persist the raw event durably (enqueue or insert), this is the only "work" you do synchronously, and it must be cheap.
4. Return 200.
5. Process the event later from the queue.

If you return 200 before verifying, a forged request gets acknowledged and an attacker learns your endpoint silently swallows anything. If you do the heavy processing (suppression, follow-up sends) in step 3, you reintroduce the timeout problem. The durable-but-cheap write in step 3 is the hinge: it is fast enough not to blow the budget, yet durable enough that a process crash after the 200 does not lose the event.

What status code should I return when?

The subtle one is the last row: returning a non-2xx for an event you simply do not care about makes the provider retry it for 24 hours for nothing. Acknowledge unknown events and discard them.

Step 2: How do I verify webhook signatures?

Always verify webhook signatures to prevent spoofing. Your webhook URL is, by necessity, a public HTTP endpoint. Anyone who discovers it could POST fake events: a forged delivered to mark a failed email as successful, or a malicious bounced to suppress a legitimate address. Signature verification proves each request genuinely came from your provider.

import crypto from 'crypto';

// Standard Webhooks scheme (standardwebhooks.com):
// HMAC-SHA256 over `${id}.${timestamp}.${rawBody}` using the shared secret.
// The secret is base64-encoded and prefixed "whsec_"; decode before signing.
// Prefer your provider's official verification helper when one exists .
// this hand-rolled version shows the mechanism and is portable.

const rawSecret = process.env.EMAIL_WEBHOOK_SECRET!;       // "whsec_<base64>"
const secretKey = Buffer.from(rawSecret.replace(/^whsec_/, ''), 'base64');

// express.raw() preserves the exact bytes the provider signed
app.post('/webhooks/email', express.raw({ type: 'application/json' }), (req, res) => {
  const id = req.header('webhook-id');
  const timestamp = req.header('webhook-timestamp');
  const signatureHeader = req.header('webhook-signature'); // space-separated "v1,<b64> v1,<b64>"
  const rawBody = (req.body as Buffer).toString('utf8');

  if (!id || !timestamp || !signatureHeader) {
    return res.status(400).send('Missing signature headers');
  }

  // 1. Replay guard: reject anything older (or newer) than 5 minutes.
  const now = Math.floor(Date.now() / 1000);
  const ts = Number(timestamp);
  if (!Number.isFinite(ts) || Math.abs(now - ts) > 300) {
    return res.status(400).send('Timestamp outside tolerance');
  }

  // 2. Recompute the expected signature over the EXACT bytes received.
  const signedContent = `${id}.${timestamp}.${rawBody}`;
  const expected = crypto.createHmac('sha256', secretKey).update(signedContent).digest('base64');

  // 3. A header can carry MULTIPLE signatures (key rotation). Accept if ANY matches,
  //    each compared in constant time.
  const candidates = signatureHeader.split(' ')
    .map((part) => part.split(',')[1])  // strip the "v1," version prefix
    .filter(Boolean);

  const ok = candidates.some((provided) => {
    const a = Buffer.from(provided, 'base64');
    const b = Buffer.from(expected, 'base64');
    return a.length === b.length && crypto.timingSafeEqual(a, b);
  });

  if (!ok) return res.status(400).send('Invalid signature');

  // Verified. Persist cheaply, then acknowledge.
  enqueueRawEvent(id, rawBody).then(
    () => res.status(200).send('OK'),
    () => res.status(503).send('Queue unavailable'), // transient -> provider retries
  );
});

How verification works: the provider signs each payload with a shared secret and sends the signature in a header. You recompute the HMAC-SHA256 over the signed content and compare it against the header in constant time with crypto.timingSafeEqual. If they do not match, reject the request with 400. The exact header names and signature encoding vary by provider, so check their docs or use their verification helper. Several operational details matter:

• Sign over the raw body. The signature is computed against the exact bytes the provider sent. If your framework parses and re-serializes the JSON before you verify, the bytes can change (key order, whitespace, Unicode escaping, trailing newline) and verification fails. The snippet above uses express.raw() to capture the original bytes; make sure you always verify against the unmodified payload. In Next.js route handlers, read await req.text() and disable body parsing; in AWS Lambda behind API Gateway, beware base64-encoded bodies (event.isBase64Encoded).
• Guard against replay with the timestamp. Providers include a timestamp header and fold it into the signed content; reject anything outside a few minutes (a ~5-minute tolerance is common) so an attacker cannot capture a valid request and replay it days later. Keep your server clock synced with NTP or valid requests can fail this check.
• Constant-time comparison only. Use crypto.timingSafeEqual, never === or Buffer.compare for the decision, and guard the length first (timingSafeEqual throws on length mismatch). A naive === leaks how many leading bytes matched via timing, which over many attempts can let an attacker forge a signature byte by byte.
• Support multiple active signatures. During secret rotation the provider may send two signatures in one header. Accept the request if any candidate verifies. This lets you rotate the secret with zero downtime: add the new secret, wait until both old and new traffic verify, then retire the old one.
• Keep the secret secret. It lives in an environment variable or a secrets manager, never in source control. Anyone with it can forge events. Rotate it if it ever leaks.

What if my provider does not use Standard Webhooks?

Not every provider uses the webhook-id/webhook-timestamp/webhook-signature triplet. The verification shape is what transfers, not the header names. Common variants you will encounter:

• Plain HMAC over the body, signature in one header. Mailgun-style: the provider sends timestamp, token, and signature, and you compute HMAC-SHA256(key, timestamp + token). Same constant-time comparison, same replay window on the timestamp.
• HMAC with the signature on the body alone. Some providers sign just the raw body and send a hex digest in a header (e.g. X-Signature). Decode hex instead of base64; everything else is identical.
• Notification-bus signatures. Amazon SES does not POST directly; it publishes to SNS, which signs each message with an X.509 certificate. Verification there means fetching the signing cert from the SigningCertURL (validate the host is an amazonaws.com SNS endpoint before fetching), verifying the message signature, and confirming the subscription. Use the AWS SDK's SNS message-validation helper rather than hand-rolling this.
• Allowlisting source IPs is a weak supplement, not a substitute. IPs change, and a leaked URL plus a spoofable source is not as strong as a cryptographic signature. Verify the signature regardless.

Whatever the scheme, the invariants are the same: verify against raw bytes, compare in constant time, enforce a timestamp window, and reject on any failure.

Step 3: How do I register the webhook URL?

Configure your webhook endpoint in your provider's dashboard or via their API. This is where you tell the provider which URL to call and which event types you want delivered there. A few practical tips:

• Register the endpoint per environment: a separate URL and secret for staging and production, so test events never hit production data.
• Subscribe only to the events you actually handle, to reduce noise and load.
• Use HTTPS, never plain HTTP, so the payload and headers cannot be read in transit.
• Put the endpoint on a stable, unauthenticated-but-signature-protected path. Do not hide it behind your app's session auth, the provider has no session. Signature verification is the auth.
• If your provider supports it, scope the subscription's source so only the provider's documented egress IPs can reach the path at your edge/CDN, as defense in depth on top of signatures.

Endpoint setup checklist

• Endpoint accepts POST and returns 2xx within 5 seconds.
• Signature is verified before the response is sent.
• Verification runs against the raw, unmodified body.
• Timestamp replay window (~5 min) is enforced.
• Comparison is constant-time and length-guarded.
• Multiple signatures are accepted during rotation.
• The webhook secret lives in an environment variable / secrets manager, not in code.
• Separate endpoints and secrets for staging and production.
• The endpoint is served over HTTPS.
• Unknown event types are acknowledged with 200, not failed.

How do I process each event type?

Once a payload is verified and pulled off the queue, route it to a handler per event type. The three below are the ones with real consequences: bounces, complaints, and deliveries.

How do I handle a bounce?

A bounce means the recipient's server refused the message. The critical distinction is hard versus soft:

async function handleBounce(event: CanonicalEvent) {
  const { email, messageId } = event;

  if (event.bounceClass === 'hard') {
    // Permanent failure - remove from all lists
    await suppressEmail(email, 'hard_bounce');
    await removeFromAllLists(email);
  } else {
    // Soft bounce - track and remove after threshold
    await incrementSoftBounce(email);
    const count = await getSoftBounceCount(email);
    if (count >= 3) {
      await suppressEmail(email, 'soft_bounce_limit');
    }
  }
  await updateEmailStatus(messageId, 'bounced');
}

The logic mirrors the two failure modes:

• A hard bounce is a permanent failure (the address does not exist, the domain is invalid). There is no point ever trying again, so you suppress immediately and remove from all lists. Continuing to mail a hard-bounced address is one of the fastest ways to damage your sender reputation, because mailbox providers read it as a sign you do not maintain your list.
• A soft bounce is a temporary failure (mailbox full, server briefly down, message too large). The address might recover, so you do not suppress on the first one. Instead you count soft bounces and suppress only after a threshold (here >= 3). This tolerance avoids discarding a good address over a transient blip, while still cutting off an address that is persistently unreachable.

Both paths ultimately feed your suppression list, which is the single source of truth checked before every send (see List Management).

What counts as a hard bounce versus a soft bounce?

A practical rule: never treat a single soft bounce as failure, and never give a hard bounce a second chance. If your provider reports a sub-type or SMTP code, log it so you can audit borderline cases later.

Read the SMTP enhanced status code, not just hard/soft

A provider's hard/soft flag is a summary. The underlying SMTP reply carries an enhanced status code (RFC 3463, the class.subject.detail triplet) that tells you why, and acting on the detail is what separates a competent bounce pipeline from a crude one. The first digit is the class: 2.x.x success, 4.x.x transient (soft), 5.x.x permanent (hard).

Authentication-failure bounces such as 5.7.26 matter especially: they mean the receiving server rejected your mail because your SPF (RFC 7208), DKIM (RFC 6376), or DMARC (RFC 7489) alignment failed. Suppressing the recipient here would be exactly wrong, the address is fine; your authentication is broken. Bounces with authentication-related codes should page your team, not prune your list. See Deliverability for how to fix the underlying records.

A common mistake: mis-bucketing "mailbox full"

Many providers report a full mailbox (5.2.2) as a permanent/hard bounce because the reply code starts with 5. A full mailbox is almost always temporary. If you suppress on the first one, you permanently lose a recipient whose inbox was briefly over quota. Override your provider's classification for known-transient 5.2.x causes and treat them as soft.

How do I handle a complaint?

A complaint means the recipient clicked "mark as spam". This is the most serious signal you can receive, and it allows no nuance, no threshold, no second chance:

async function handleComplaint(event: CanonicalEvent) {
  const { email } = event;

  // Immediate suppression - no exceptions
  await suppressEmail(email, 'complaint');
  await removeFromAllLists(email);
  await logComplaint(event); // For analysis
}

Why the zero-tolerance treatment: a spam complaint is a direct, explicit statement that the recipient does not want your email. Sending them anything more, even a "sorry to see you go" message, risks another complaint. High complaint rates are weighted heavily by mailbox providers and are among the quickest routes to the spam folder for all your recipients, not just the one who complained.

The thresholds are not arbitrary. The Gmail, Yahoo, and Microsoft bulk-sender requirements that took effect in February 2024 make this explicit for anyone sending to those providers at volume (roughly 5,000+ messages/day to a given provider counts as a bulk sender): you must keep your spam-complaint rate, as measured by Google's Postmaster Tools, below 0.3%, and you should aim well under that, a practical target is under 0.1% (one complaint per thousand sent). Cross 0.3% and Gmail actively throttles or junks your mail. Those same requirements also mandate authenticated mail (SPF + DKIM + DMARC) and one-click unsubscribe per RFC 8058 (a List-Unsubscribe header plus List-Unsubscribe-Post: List-Unsubscribe=One-Click, handled by a POST). Complaints and unsubscribes are two sides of the same coin: a frictionless unsubscribe is the pressure-release valve that keeps the complaint rate down. See Compliance and Sending Reliability for the header mechanics.

So you suppress immediately and unconditionally, remove from every list, and log the complaint so you can analyze which campaign or message is generating complaints and fix the root cause.

Complaints arrive late and sometimes anonymized

Two operational realities to design for:

• Latency. Complaint feedback (often via ARF feedback loops, RFC 5965) can lag the send by hours or days. Your complaint rate for a campaign is not final until that window passes. Do not declare a send "clean" the moment delivery completes.
• Anonymized recipients. Some feedback loops redact the complaining address (Microsoft's, historically). When the address is missing or hashed, you cannot suppress by address directly. This is one reason to embed a per-recipient identifier in a custom header or in your message metadata at send time, so a redacted complaint can still be traced back to a recipient via the provider message ID.

How do I handle a delivery confirmation?

async function handleDelivered(event: CanonicalEvent) {
  const { messageId } = event;
  await updateEmailStatus(messageId, 'delivered');
}

This is the simplest handler, and it is where the message ID you stored at send time earns its keep: you look up the original send record by that ID and mark it delivered. Now your system can answer "did this email actually arrive?" with authority. That is useful for support, for showing delivery status in a UI, and for spotting messages that were sent but never delivered, which is a sign of a silent delivery problem.

Beware out-of-order delivery

Webhooks are not guaranteed to arrive in the order the events occurred. Retries, parallel delivery workers, and queue rebalancing all reorder events. You can receive delivered after you already received opened for the same message, or a retry of sent after delivered. Make status transitions monotonic: store each event, and compute the current status as the furthest-along state ever seen for that message, never by blindly overwriting with whatever event arrived last. A delivered message that later receives a stale sent retry must stay delivered.

// Rank states so a late, out-of-order event can never regress status.
const RANK: Record<string, number> = {
  queued: 0, sent: 1, delivered: 2, opened: 3, clicked: 4,
};
// Terminal failure states are tracked on a separate axis (bounced/complained),
// because a delivered message can still become complained.

async function advanceStatus(messageId: string, next: string) {
  const current = await getStatus(messageId);
  if ((RANK[next] ?? -1) > (RANK[current] ?? -1)) {
    await setStatus(messageId, next);
  }
}

Why does my webhook handler have to be idempotent?

Webhooks may be sent multiple times. This is not a bug. It is a direct consequence of the "acknowledge first, retry on non-2xx" model. If your 200 response is delayed, lost in transit, or your server restarts mid-processing, the provider assumes failure and redelivers the same event. If your handler is not idempotent, that redelivery double-counts an open, double-suppresses, or sends a follow-up twice.

It is worth being precise: most providers guarantee at-least-once delivery, not exactly-once. Exactly-once is something you construct on top of at-least-once by deduplicating. Anyone who tells you their webhooks are exactly-once is describing a goal, not a network guarantee.

The fix is to deduplicate on the event ID, exactly mirroring the idempotency-key pattern from the sending side:

async function processWebhook(event: CanonicalEvent) {
  const eventId = event.id;

  // Check if already processed
  if (await isEventProcessed(eventId)) {
    return; // Skip duplicate
  }

  // Process event
  await handleEvent(event);

  // Mark as processed
  await markEventProcessed(eventId);
}

Each event carries a stable event.id. Before doing any work, check whether you have already processed that ID. If so, return early. Otherwise process it, then record the ID as processed.

What if my provider does not send a stable event ID?

Some providers omit a unique event ID, or send one that changes on retry (which makes it useless for dedup). When that happens, synthesize a deterministic key from fields that are stable across retries of the same event:

import crypto from 'crypto';

// Build a stable dedup key when the provider gives no reliable event id.
// Use fields that are identical across retries of the same logical event.
function dedupeKey(e: CanonicalEvent): string {
  const basis = [e.messageId, e.type, e.email, e.occurredAt.toISOString()].join('|');
  return crypto.createHash('sha256').update(basis).digest('hex');
}

Pick fields that uniquely identify the event but do not vary between retries (the occurrence timestamp, not the delivery timestamp). Hash them to a fixed-length key and use that as your idempotency key.

What is the subtle bug to watch for in idempotent processing?

Ideally the "process" and "mark processed" steps happen in the same database transaction, so a crash between them cannot leave an event in a broken state:

• Handled but unmarked: the next retry reprocesses it, which is exactly the double-processing you were trying to avoid.
• Marked but unhandled: the event is silently dropped and never acted on.

Wrapping both steps in one transaction (or using an upsert on the event ID as the first write) removes both failure modes. The most robust pattern is to make the dedup insert the first write and let a unique-constraint violation short-circuit the rest:

async function processOnce(event: CanonicalEvent) {
  // INSERT ... ON CONFLICT DO NOTHING. If the row already exists, rowCount === 0
  // and we know another delivery already claimed this event.
  const { rowCount } = await db.query(
    `INSERT INTO processed_events (event_id, created_at)
     VALUES ($1, now()) ON CONFLICT (event_id) DO NOTHING`,
    [event.id],
  );
  if (rowCount === 0) return; // duplicate, already claimed

  // Do the real work in the SAME transaction as the claim where possible,
  // so a crash here rolls back the claim and the event is retried cleanly.
  await handleEvent(event);
}

Idempotent processing is what makes the provider's aggressive retries safe rather than dangerous. Store processed event IDs with a retention window (for example 30 days) so the dedup table does not grow without bound; the provider stops retrying long before that. A simple scheduled job that deletes rows older than the retention window keeps it bounded.

What happens when my endpoint returns an error?

If your endpoint returns non-2xx (or times out), webhooks will retry with exponential backoff:

• Retry 1: about 30 seconds
• Retry 2: about 1 minute
• Retry 3: about 5 minutes
• (continues, with growing gaps, for about 24 hours)

The exact schedule and total window vary by provider, some retry for 24 hours, some for up to three days, some cap the number of attempts rather than the duration. Read your provider's documented policy and design for the shortest window you might face, not the most generous.

This is the provider being a good citizen on your behalf: a transient outage on your side will not lose events, because the provider keeps trying for roughly a day. But it has direct consequences for how you build the endpoint.

• It is the reason idempotency is mandatory. Every retry redelivers the same event.
• A persistently broken endpoint will eventually give up. You must monitor for repeated webhook failures and fix them within the retry window or you will lose those events permanently.

Note the symmetry with the sending side from the previous chapter: both directions use exponential backoff, both retry only transient failures, and both stop after a bounded window. See Sending Reliability.

What happens after the provider gives up? Reconcile.

The retry window is finite, so a long enough outage will drop events for good. The defense is reconciliation: a periodic sweep that compares your view of the world against the provider's and fills gaps. Two practical mechanisms:

• Provider event export / search API. Many providers expose an API to list events in a time range. Run a job that, for any message stuck in sent past a reasonable delivery SLA, queries the provider's event history and applies anything you missed.
• Most providers also keep a dashboard log and an endpoint health view. Most providers expose a webhook delivery log showing failed deliveries and a manual "replay" button. After fixing an outage, replay the failed deliveries from that window. Most providers also publish an ingestion guide or reference implementation, check your provider's webhook docs.

Reconciliation is also your safety net for the at-least-once-but-sometimes-zero reality of any network system. Treat webhooks as the fast path and reconciliation as the correctness backstop.

Disabled endpoints and circuit breakers

Some providers automatically disable a webhook endpoint that fails for an extended period (for example, a high error rate over many consecutive hours), and stop sending to it entirely until you re-enable it. This is a sharp edge: a deploy that breaks your endpoint for a few hours can get your webhook switched off, after which you silently receive nothing even once you fix the code. Monitor the endpoint's enabled/health state in the provider dashboard, and alert if delivery volume drops to zero unexpectedly.

What are the best practices for a production webhook pipeline?

• Return 200 quickly: process asynchronously to avoid timeouts.
• Be idempotent: handle duplicate deliveries gracefully.
• Log everything: store raw events for debugging.
• Alert on failures: monitor webhook processing errors.
• Queue for processing: use a job queue for complex handling.

Each of these maps to a problem you will otherwise hit in production:

• Return 200 quickly prevents the provider from timing out and triggering the retry storm described above.
• Be idempotent keeps those retries harmless.
• Log everything means storing the raw event payloads. When you need to debug "why was this address suppressed?" three weeks later, the raw event is your ground truth, and it lets you replay processing if a handler had a bug.
• Alert on failures matters because a webhook pipeline failing silently means your suppression list quietly rots. Monitor processing errors and page on sustained failure.
• Queue for processing means that for anything beyond a trivial status update, enqueue the verified event to a durable job queue. This is the same durability argument as the sending side: it survives restarts and decouples receipt from processing.

What should I monitor and alert on?

A webhook pipeline fails quietly, so the alerts you wire up are what keep it honest. The high-value signals:

Send permanently-unprocessable events to a dead-letter queue rather than retrying them forever. A payload your handler cannot parse will never succeed; isolate it for inspection instead of letting it block the pipeline.

What are the most common webhook mistakes?

• Doing real work before sending 200, which causes timeouts and a retry storm.
• Returning 200 before verifying the signature, which acknowledges forged events.
• Skipping signature verification entirely, which leaves the endpoint open to forged events.
• Verifying against a re-serialized body instead of the raw bytes, so valid signatures fail.
• Comparing signatures with === instead of a constant-time function.
• Omitting the timestamp replay check, leaving captured requests replayable.
• Forgetting idempotency, so retries double-count or double-suppress.
• Treating a single soft bounce as permanent, or giving a hard bounce a second chance.
• Suppressing on an authentication-failure bounce (5.7.26) instead of fixing your SPF/DKIM/DMARC.
• Mis-bucketing "mailbox full" as a hard bounce.
• Overwriting status with the last event received instead of the furthest-along state (out-of-order bug).
• Failing unhandled event types with non-2xx, triggering 24 hours of pointless retries.
• Not monitoring the pipeline, so failures rot the suppression list unnoticed.

How do I test webhooks before going live?

Webhooks are awkward to test because the provider needs to reach a public URL, but your code runs on localhost.

For local development, use a tunneling tool to expose localhost. ngrok is the common choice; alternatives include cloudflared (Cloudflare Tunnel) and your provider's own CLI if it offers one.

ngrok http 3000
# Use the printed https://<random>.ngrok-free.app URL as your webhook endpoint

The tunnel creates a public address routed to your local port, giving you a temporary public URL you can register as the webhook endpoint. Now real events from the provider reach your local handler, where you can set breakpoints and inspect payloads. Remember the tunnel URL changes each restart on free tiers, so re-register it when it does (a reserved/static domain avoids this).

Generating a valid signed request without the provider

You do not always want to depend on the provider to test. Because the signature scheme is just HMAC over id.timestamp.body, you can mint a correctly signed request locally and POST it with curl, which lets you test verification, idempotency, and each handler deterministically:

#!/usr/bin/env bash
# Sign a payload the Standard Webhooks way and POST it to the local endpoint.
SECRET_B64="${EMAIL_WEBHOOK_SECRET#whsec_}"   # base64 part of "whsec_..."
ID="msg_$(date +%s)"
TS="$(date +%s)"
BODY='{"type":"email.bounced","data":{"email":"nobody@example.com","bounce_type":"hard"}}'

SIGNED="${ID}.${TS}.${BODY}"
SIG=$(printf '%s' "$SIGNED" \
  | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$(printf '%s' "$SECRET_B64" | base64 -d | xxd -p -c 256)" -binary \
  | base64)

curl -sS -X POST http://localhost:3000/webhooks/email \
  -H "content-type: application/json" \
  -H "webhook-id: $ID" \
  -H "webhook-timestamp: $TS" \
  -H "webhook-signature: v1,$SIG" \
  --data-raw "$BODY"

To verify handling, send test events through your provider's dashboard or with a script like the one above. Do not just test the happy path. Deliberately exercise each handler and each failure mode:

• Trigger a hard bounce and confirm the address is suppressed.
• Trigger a soft bounce three times and confirm suppression only on the third.
• Trigger a complaint and confirm immediate removal.
• Send the same event twice and confirm your idempotency check skips the duplicate.
• Send a payload with a bad signature and confirm you reject it with 400.
• Send a payload with a stale timestamp and confirm the replay guard rejects it.
• Send opened before delivered for one message and confirm status does not regress.
• Send a 5.7.26 authentication-failure bounce and confirm you alert rather than suppress.
• Send a slow handler into its timeout path and confirm a retry is processed harmlessly.

Processing and testing checklist

• Each event type has a dedicated, tested handler.
• Hard bounces and complaints suppress immediately; soft bounces use a threshold.
• "Mailbox full" and authentication-failure bounces are classified correctly.
• Processing deduplicates on event.id (or a synthesized stable key).
• "Process" and "mark processed" are atomic where possible.
• Status transitions are monotonic (no regression from out-of-order events).
• Raw event payloads are stored for debugging and replay.
• Webhook processing failures and zero-volume conditions trigger alerts.
• Duplicate-delivery, bad-signature, and replay cases are tested explicitly.

Should I store the full webhook history, or just the latest status?

Store the full event history, not just the latest status. A typical schema is an append-only email_events table keyed by the provider's event.id, storing the event type, the referenced message ID, a timestamp, and the raw JSON payload, with your current per-message status derived from it (or kept in a separate table updated as events arrive). Most providers also publish a reference implementation or ingestion guide; check your provider's webhook docs.

A workable schema:

-- Append-only event log: ground truth, never updated in place.
CREATE TABLE email_events (
  event_id     TEXT PRIMARY KEY,        -- provider id or synthesized dedup key
  message_id   TEXT NOT NULL,           -- your stored provider message id
  type         TEXT NOT NULL,           -- canonical: sent/delivered/bounced/...
  email        TEXT NOT NULL,
  occurred_at  TIMESTAMPTZ NOT NULL,
  received_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  raw          JSONB NOT NULL           -- verbatim provider payload
);
CREATE INDEX ON email_events (message_id);
CREATE INDEX ON email_events (email, type);

-- Derived current status per message (rebuildable from email_events at any time).
CREATE TABLE email_status (
  message_id   TEXT PRIMARY KEY,
  status       TEXT NOT NULL,
  bounced      BOOLEAN NOT NULL DEFAULT false,
  complained   BOOLEAN NOT NULL DEFAULT false,
  updated_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

Storing the full event history gives you an audit trail, the ability to recompute metrics after a logic change, and the raw material to debug deliverability trends over time. If you only keep the latest status, you lose the ability to answer "when did this address start bouncing?" or to rebuild a metric you defined incorrectly the first time. The append-only log plus a derived status table is the best of both: the log is authoritative and replayable, the status table is fast to query, and the status table can always be regenerated from the log.

How long should I retain raw events?

Balance debuggability, cost, and privacy. Raw payloads contain recipient email addresses (personal data under GDPR), so retention is also a compliance decision, not just a storage one, see Compliance. A common pattern: keep full raw payloads hot for 30–90 days for debugging and replay, then either delete them or down-sample to the structured columns (dropping the raw blob) for long-term metrics. Keep your dedup window (e.g. 30 days) at least as long as the provider's maximum retry window so a late retry still deduplicates.

What should I read next?

• List Management: what to do with bounce and complaint data.
• Sending Reliability: retry logic when sends fail.
• Deliverability: how bounces and complaints affect sender reputation.
• Compliance: retention of recipient data and one-click unsubscribe obligations.
• Index