Sending Reliability

Sending an email looks like one function call, but underneath it is a distributed-systems problem. An unreliable network sits between your application and the email provider, and either side can fail, slow down, or lose its memory of what happened. This chapter answers how to make that uncertain world behave predictably: every email that should be sent is sent exactly once, transient failures recover automatically, and permanent failures are caught, logged, and surfaced to a human before they cause damage.

The four pillars are idempotency (never send a duplicate), retry logic (recover from transient failures), error handling (react correctly to each failure class), and queuing (survive crashes and decouple sending from the request path). Timeouts tie them together. The examples use a generic emailClient wrapper, but every pattern is provider-agnostic. The same principles apply to Amazon SES, Postmark, SendGrid, Mailgun, Resend, Brevo, SparkPost, Mandrill, or your own SMTP relay.

This chapter is about the mechanics of getting bytes from your application to the provider reliably. It is the operational counterpart to two adjacent topics: Transactional Emails covers what those emails should contain and when to send them, and Webhooks & Events covers what happens after the provider accepts a message (delivery, bounce, complaint). A send accepted by the provider is not the same as a message delivered to the inbox; this chapter ends at provider acceptance, and the webhook chapter picks up from there.

Quick answers:

• Why does email need reliability engineering at all? Because a duplicate, dropped, or storm-retried email costs trust, support tickets, and sender reputation.
• How do I stop sending duplicates? Use an idempotency key derived from a stable business event.
• Which errors should I retry? 5xx, 429, timeouts, connection resets, and DNS failures. Never a plain 4xx.
• How long should I wait between retries? Exponential backoff (1s, 2s, 4s, 8s) capped at 30s, with jitter.
• When do I need a queue? For anything the user is waiting on or anything legally required.
• What timeout should I set? 10 to 30 seconds per API call.
• What is the one rule that ties it all together? The worst failure is the silent one, every code path must log, queue, or alert.

Why does email need reliability engineering at all?

Because the failure modes are not cosmetic, they cost trust, support load, and deliverability. Reliability is not a nice-to-have for transactional email. It is the difference between a system users trust and one they don't.

Three concrete examples of what goes wrong without it:

• A duplicate "Your password has been reset" email erodes trust and can read as a security signal to the recipient.
• A silently dropped order confirmation generates a support ticket and a confused customer.
• A retry storm during an outage can get your sending IP throttled, which then hurts every other email you send.

The rest of this chapter is the toolkit that prevents each of these.

Why is "the request returned 200" not the same as "the email arrived"?

Sending email has at least four distinct success boundaries, and conflating them is the root of most reliability bugs. Treat them as separate states:

This chapter's job is to make the first two boundaries reliable: every message that should be accepted by the provider is accepted exactly once. The later boundaries are out of your synchronous control and are observed asynchronously via webhooks. Designing as if a 200 means "the user got the email" will mislead every incident investigation you ever run.

What does "at scale" actually mean for failure rates?

Failures that feel like freak events at low volume are routine at volume. If a single send has a 99.9% success rate (a 0.1% transient failure rate, which is optimistic for any cross-network call), then at 1,000,000 sends per day you will see roughly 1,000 transient failures every day. Without automatic recovery, that is 1,000 lost emails per day. The entire point of this chapter is to convert that 0.1% from "lost messages" into "automatically recovered messages, invisible to the user." Reliability engineering is not paranoia; it is arithmetic.

How do I make sure I never send the same email twice?

Use an idempotency key. Idempotency means an operation can be performed many times and produce the same result as performing it once. For email, an idempotent send guarantees that retrying a request, for any reason, never produces a second email in the recipient's inbox.

What exactly goes wrong without idempotency?

The classic ambiguous failure. Network issues, timeouts, or server errors can leave you uncertain whether an email was actually sent:

1. Your app calls the email API.
2. The provider receives the request, sends the email, and starts writing the response.
3. The network connection drops before the response reaches you.
4. Your app sees a timeout and assumes the send failed.

From your side, this is indistinguishable from a request that never arrived. If you retry naively, you send the email twice. The recipient gets two identical messages, and for a high-stakes email like a payment receipt or a one-time login code, that is both confusing and a potential security signal.

The uncomfortable truth: you can never be 100% sure a request succeeded from the client side alone. This is the two generals problem applied to email, no number of acknowledgements can give both sides certainty in the presence of a lossy channel. Any system that retries (and reliable systems must retry) needs a way to make retries safe. That mechanism is idempotency. Idempotency does not eliminate the uncertainty; it makes the uncertainty harmless by pushing the deduplication decision to the one place that has authoritative knowledge, the provider.

How does an idempotency key work?

You send a unique key with each request. The server records the key alongside the result of the first request. If the same key arrives again, the server returns the original response instead of performing the action a second time.

// Generate deterministic key based on the business event
const idempotencyKey = `password-reset-${userId}-${resetRequestId}`;

await emailClient.send({
  from: 'noreply@example.com',
  to: user.email,
  subject: 'Reset your password',
  html: emailHtml,
}, {
  headers: {
    'Idempotency-Key': idempotencyKey
  }
});

The key insight is that the server enforces uniqueness. Your retry logic can be brutally simple, just send the same request with the same key again, and correctness is guaranteed on the provider side. This is far more robust than trying to track "did this succeed?" in your own database, because it closes the exact race condition described above (the response lost in transit).

What happens on the server side when a key repeats?

Understanding the provider's state machine tells you exactly what guarantees you do and do not get. A correct idempotent endpoint behaves like this:

1. First request with key K arrives. The provider atomically reserves K (typically INSERT ... ON CONFLICT DO NOTHING, or a SETNX in a key-value store), then performs the send and stores the response body and status under K.
2. A repeat of K while the first is still in flight returns either a 409 Conflict ("a request with this key is in progress") or blocks until the first finishes. Your retry logic must treat 409 as "the original is winning, back off and re-check," not as a hard failure.
3. A repeat of K after the first completed returns the stored response, same status code, same message ID, without sending again.

Two consequences follow. First, the key must be reserved before the side effect, or two concurrent requests can both pass the check and both send. Second, the dedup is scoped to the request payload only as far as the provider chooses; some providers reject a repeated key whose body differs (422), others silently ignore the new body and replay the old result. Never rely on changing the body under a reused key, generate a new key when the content legitimately changes.

How should I generate the key?

The value of idempotency depends entirely on how you generate the key. A good key is stable across every retry of the same logical send, and different for every distinct send.

How to choose:

• Event-based keys are the gold standard. If your business rule is "exactly one order confirmation per order," then order-confirm-${orderId} encodes that rule directly in the key. Even if two separate code paths or two concurrent workers try to send the confirmation, the provider deduplicates them. This makes the key meaningful and self-documenting.
• Request-scoped keys suit cases where the same user can legitimately trigger many sends of the same type, but each individual request should produce one email. A password reset is the canonical example: the user may request several resets, and each gets its own resetRequestId, so each is a distinct send, but a retry within one request reuses the same key.
• Composite keys matter in multi-tenant systems. Namespacing the key with a tenant identifier prevents a collision where two tenants happen to share an orderId from independent counters. A bare order-confirm-1001 is dangerous across tenants; tenant-42:order-confirm-1001 is safe.
• UUIDs are the fallback when no natural identifier exists. The critical rule: generate the UUID once, store it, and reuse it on every retry. A UUID generated fresh on each attempt provides zero protection. It is no better than no key at all.

Best practice: use deterministic keys based on the business event. If you retry the same logical send, the same key must be generated. Avoid Date.now() or random values generated fresh on each attempt, because both change between retries and defeat the entire mechanism. A good mental test: "If my process crashes and restarts mid-send, will my code regenerate the identical key?" If the answer is no, your key is wrong.

Worked example: deriving a stable key from an event

When you have no obvious single identifier, hash the stable parts of the event with Node's standard crypto, no external dependency, fully deterministic:

import { createHash } from 'node:crypto';

// Deterministic key from the immutable facts of the send.
// Same inputs => same key, on every retry and after any crash.
function idempotencyKeyFor(event: {
  type: string;        // e.g. "order.confirmation"
  tenantId: string;
  recipient: string;
  resourceId: string;  // e.g. orderId, invoiceId
}): string {
  const canonical = [
    event.type,
    event.tenantId,
    event.recipient.trim().toLowerCase(),
    event.resourceId,
  ].join('|');

  // 256-bit hash, hex-encoded. Stable, collision-resistant, opaque.
  return createHash('sha256').update(canonical).digest('hex');
}

Normalizing the recipient (trim().toLowerCase()) is deliberate: although the SMTP local part is technically case-sensitive (RFC 5321 §2.4), in practice every major mailbox provider treats User@Example.com and user@example.com as the same mailbox, and you do not want a casing difference to look like a distinct send. The domain part is case-insensitive by definition, so lowercasing it is always safe.

What is a common idempotency mistake to avoid?

Treating the key as a per-attempt token rather than a per-send token. The most frequent errors:

• Generating a fresh random value or timestamp inside the retry loop, so every attempt carries a different key. The provider sees each attempt as a brand new send and dedup does nothing.
• Not persisting the key with a queued job. A message that sits in a dead-letter queue and is replayed later loses the protection if the key was only held in memory.
• Including volatile data in the key. If your key incorporates a rendered timestamp, a request ID, or a load-balancer-assigned trace ID, it changes on retry. Keys must be built only from immutable facts of the business event.
• Reusing a key across genuinely different content. If you reuse order-confirm-1001 to also send the shipping notification for order 1001, the provider may replay the confirmation and silently drop the shipping email. One key per distinct logical message.
• Assuming idempotency replaces your own dedup. The provider's key cache is bounded in time (see below). For guarantees beyond that window you still need an application-level "already sent" record.

How long does an idempotency key stay valid?

Idempotency keys are typically cached for 24 hours, though the exact retention varies by provider, some keep keys for as little as a few minutes, others for several days, and a few let you configure it. Retries within this window return the original response. After expiration, the same key triggers a new send, so complete your retry logic well within the provider's window. In practice this is rarely a constraint (most retries finish within minutes), but it matters for long-lived queues: a message that sits in a dead-letter queue for two days and is then replayed with its original key will send a fresh email, because the provider has forgotten the key. Design your queue retention with the provider's window in mind, and for anything that must be exactly-once beyond that window, keep your own sent_emails(idempotency_key UNIQUE) table as the durable backstop.

Idempotency checklist

• Every send carries an Idempotency-Key header.
• The key is derived from a stable business identifier, not a timestamp or fresh random value.
• The same logical send regenerates the identical key after a crash or restart.
• Keys are namespaced per tenant in multi-tenant systems.
• Retry windows are well under the provider's key-retention window, or the key is persisted with the queued job.
• Distinct sends (e.g. two separate password resets, or a confirmation vs a shipping notice) produce distinct keys.
• A 409 Conflict on a repeated key is treated as "original in flight," not a hard error.
• An application-level unique constraint backs up the provider's bounded cache for exactly-once-forever guarantees.

When should I retry a failed send?

Only when the failure is transient. Transient failures are normal at scale. A reliable sender treats them as expected events and recovers automatically, but it must retry intelligently, distinguishing failures worth retrying from failures that will never succeed, and spacing retries to avoid making an outage worse.

Which errors are worth retrying?

Not every error should be retried. Retrying a malformed request just wastes time and quota; retrying a transient server error usually succeeds. Classify the failure first:

The dividing line is whose problem it is. A 4xx (except 429 and 408) means your request is wrong, an invalid email address, a missing field, an unverified domain. Retrying an unchanged bad request will fail identically every time and only burns resources. A 5xx, a 429, a timeout, or a DNS hiccup means something on the path is temporarily unhealthy, exactly the situation retries exist for.

Why are idempotency and retries inseparable?

There is a category of error, ECONNRESET, a dropped connection, a timeout after the request body was fully sent, where you genuinely cannot know whether the side effect happened. These are precisely the cases that must be retried for reliability and must not duplicate. You cannot have one safely without the other. The rule is: never enable retries on a code path that does not carry an idempotency key. A retry loop without idempotency is not a reliability feature; it is a duplicate-email generator. This is why the idempotency section comes first in this chapter, it is the precondition for everything that follows.

How should I space out retries?

When you do retry, do not hammer the server. Wait progressively longer between attempts (exponential backoff), and add a small random offset (jitter) so that many clients failing at the same instant don't all retry in lockstep.

async function sendWithRetry(emailData, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await emailClient.send(emailData);
    } catch (error) {
      if (!isRetryable(error) || attempt === maxRetries - 1) {
        throw error;
      }
      const delay = Math.min(1000 * Math.pow(2, attempt), 30000);
      await sleep(delay + Math.random() * 1000); // Add jitter
    }
  }
}

function isRetryable(error) {
  return error.statusCode >= 500 ||
         error.statusCode === 429 ||
         error.statusCode === 408 ||
         error.code === 'ETIMEDOUT' ||
         error.code === 'ECONNRESET' ||
         error.code === 'ECONNREFUSED' ||
         error.code === 'EAI_AGAIN';
}

Read the logic carefully, because the details are what make it correct:

• if (!isRetryable(error) || attempt === maxRetries - 1) throw error; packs two exit conditions in one line. If the error is not retryable, give up immediately (no point waiting). If it is retryable but this was the last attempt, throw so the caller can queue, log, or alert. Never swallow the final failure silently.
• Math.min(1000 * Math.pow(2, attempt), 30000) is the exponential part. Delay doubles each attempt (1000 * 2^attempt), and Math.min(..., 30000) caps the delay at 30 seconds so backoff never grows absurdly long.
• + Math.random() * 1000 adds jitter. Without it, every client that failed at the same moment retries at the same moment, producing a synchronized spike known as the thundering herd. The random offset spreads retries across a window and smooths load on a recovering server.

Backoff schedule: 1s, then 2s, then 4s, then 8s (with jitter to prevent thundering herd). For the default maxRetries = 3 only the first three delays apply; the schedule shows how it continues as you raise the limit, always respecting the 30-second cap.

Which jitter strategy should I use?

The simple + Math.random() * 1000 above adds bounded jitter on top of the full backoff, which is fine for in-process retries. For high-fan-out systems (many workers retrying against one provider after a shared outage), full jitter spreads load more evenly and is the recommended default in distributed systems literature:

// Full jitter: a random point in [0, exponential cap].
function fullJitterDelay(attempt: number, base = 1000, cap = 30000): number {
  const exp = Math.min(cap, base * 2 ** attempt);
  return Math.random() * exp;
}

For a synchronous request path, equal jitter is plenty. For a fleet of background workers draining a queue, prefer full jitter so a recovering provider sees a smooth ramp rather than a wall of synchronized retries.

Should I respect the Retry-After header?

Yes, always prefer it. When a 429 (or sometimes a 503) response includes a Retry-After header, use it over your computed backoff. The server is telling you exactly how long its rate-limit window is. Honoring it resolves the situation faster and signals that you are a well-behaved client, which protects your reputation with the provider.

Retry-After comes in two formats per the HTTP spec, and you must handle both, delta-seconds and an HTTP-date:

// Retry-After is either "120" (seconds) or "Wed, 21 Oct 2026 07:28:00 GMT".
function parseRetryAfter(headerValue: string | null): number | null {
  if (!headerValue) return null;

  // Delta-seconds form.
  const seconds = Number(headerValue);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);

  // HTTP-date form.
  const when = Date.parse(headerValue);
  if (!Number.isNaN(when)) return Math.max(0, when - Date.now());

  return null;
}

When Retry-After is present, wait max(parseRetryAfter(...), computedBackoff), never less than the server asked for, and never undercut your own backoff floor either.

How many times should I retry before giving up?

Three attempts is a sane default for synchronous code paths, which is why maxRetries = 3 above. Beyond a few in-process attempts you are better off handing the email to a queue (covered below) so retries continue independently of the user's request. Keep total in-process retry time short, ideally under the request timeout, so you are never holding a user behind a spinner while you back off.

There are two distinct retry budgets, and confusing them causes either lost emails or stuck users:

• In-process budget: 2–3 fast attempts, bounded by the HTTP request timeout. Its job is to paper over a single transient blip without the user noticing. If it exhausts, hand off to the queue, do not block.
• Queue budget: many attempts over minutes-to-hours, with longer backoff (for example 1m, 5m, 30m, 2h). Its job is to survive a provider outage. When it exhausts, the message goes to a dead-letter queue and pages a human.

Should the retrying side ever stop trying as a fleet?

Yes, a per-request retry loop is not enough protection during a full provider outage. If thousands of workers each retry three times against a dead endpoint, you hammer it the moment it recovers. Wrap the provider call in a circuit breaker:

// Minimal circuit breaker: trip open after N consecutive failures,
// reject fast while open, allow a trial request after a cooldown.
class CircuitBreaker {
  private failures = 0;
  private openedAt: number | null = null;

  constructor(
    private readonly threshold = 5,
    private readonly cooldownMs = 30_000,
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.openedAt !== null) {
      if (Date.now() - this.openedAt < this.cooldownMs) {
        throw new Error('circuit_open'); // fail fast, queue instead
      }
      this.openedAt = null; // half-open: allow one trial
    }
    try {
      const result = await fn();
      this.failures = 0;
      return result;
    } catch (err) {
      if (++this.failures >= this.threshold) this.openedAt = Date.now();
      throw err;
    }
  }
}

While the breaker is open, route sends straight to the queue rather than burning retries against an endpoint you already know is down. This converts a provider outage from "every request slow and failing" into "requests fail fast and recover automatically when the provider returns."

Retry checklist

• Errors are classified before retrying. 4xx (except 429/408) is never retried.
• Retries only run on code paths that carry an idempotency key.
• Backoff is exponential and capped (here, at 30s).
• Jitter is added to every delay (full jitter for high-fan-out workers).
• The final failure is re-thrown, not swallowed.
• Retry-After from 429/503 responses is honored, parsing both seconds and HTTP-date forms.
• In-process and queue retry budgets are distinct.
• A circuit breaker prevents fleet-wide hammering during a provider outage.

How do I react correctly to each kind of error?

Retrying is only half of failure handling. The other half is reacting correctly to each kind of error: fixing what is fixable, retrying what is transient, and escalating what is critical.

What do the common error codes mean and what should I do?

A few of these deserve special attention because they are commonly misdiagnosed:

• 401 vs 403. A 401 means the API key is missing or invalid, authentication failed. A 403 means you are authenticated but not allowed to do this, most often because the sending domain isn't verified or the key lacks permission. Don't rotate your key when the real fix is verifying your domain.
• 400 vs 422. Both indicate a bad request, but 400 typically signals a malformed payload (unparseable body, wrong content type) while 422 signals a well-formed but semantically invalid request (a syntactically valid field with an unacceptable value). The fix differs: 400 means check how you serialize; 422 means check the data itself.
• 413 is not retryable as-is. A payload-too-large error means the attachment or HTML body exceeds the provider's per-message limit. Retrying the identical request fails identically; you must shrink the message (host large attachments behind a link) before it can succeed.
• 429 and 408 are the only retryable codes in the 4xx range. Everything else there is your bug to fix; 429 means "slow down," 408 means "the server gave up waiting."

How do API-level errors differ from SMTP-level errors?

If you send via raw SMTP rather than an HTTP API, you deal with SMTP reply codes directly, and the same transient-vs-permanent split applies. Map them the same way you map HTTP codes:

Note the inversion relative to HTTP: in SMTP, a 4xx is transient ("try again later") and a 5xx is permanent ("rejected, do not retry"). A 550 is the SMTP equivalent of a hard bounce, the address is bad or you are blocked, and retrying it damages your reputation. Feed 550/554 outcomes into suppression exactly as you would a hard-bounce webhook (see List Management).

What does a correct error-handling branch look like?

try {
  const result = await emailClient.send(emailData);
  await logSuccess(result.id, emailData);
} catch (error) {
  if (error.statusCode === 429 || error.statusCode === 503) {
    await queueForRetry(emailData, parseRetryAfter(error.retryAfter));
  } else if (error.statusCode === 408 || error.statusCode >= 500) {
    await queueForRetry(emailData);
  } else if (isHardBounce(error)) {
    await suppressRecipient(emailData.to, error);   // 550-class, invalid address
    await logFailure(error, emailData);
  } else {
    await logFailure(error, emailData);
    await alertOnCriticalEmail(emailData); // For password resets, etc.
  }
}

This pattern encodes four distinct outcomes:

1. Success: log the provider's message ID. That ID is your join key. It links this send to the delivery, bounce, and complaint webhook events you'll receive later (see Webhooks & Events). Without it, you cannot correlate a future bounce back to the originating send.
2. Transient failure (429, 503, 408, or 5xx): hand the email to a queue for retry rather than blocking the current request. For 429/503, pass the parsed Retry-After so the queue waits the right amount of time.
3. Hard bounce / invalid recipient: the address is bad. Retrying never helps and repeatedly sending to invalid addresses wrecks your sender reputation, so suppress the recipient immediately and record why.
4. Permanent failure (other 4xx): retrying is pointless, so log the failure for diagnosis and, for critical emails, alert a human. A failed marketing newsletter can wait; a failed password reset or payment receipt is a real incident. The alertOnCriticalEmail call is what turns a silent drop into an actionable page.

Design principle: the worst failure is the silent one. Every branch above produces a durable record, a success log, a queued job, a suppression entry, or a failure log plus alert. Nothing falls through the cracks.

How do I keep error logs useful without leaking PII?

Email payloads contain personal data (recipient addresses, names, sometimes tokens). Logs are read by people and shipped to third-party log systems, so be deliberate:

• Log the provider message ID, the idempotency key, the status code, and the error class, these are what you need to debug.
• Hash or truncate the recipient address in logs (u***@example.com) rather than storing it in plaintext, to honor data-minimization expectations under regimes covered in Compliance.
• Never log one-time tokens, reset links, or full HTML bodies. If a reset link lands in your log aggregator, it is a credential leak.
• Attach a correlation ID (request ID + idempotency key) so a single logical send is traceable across the request log, the queue, and the webhook events.

When do I need a queue instead of sending inline?

For critical emails, use a queue to ensure delivery even if the initial send fails. A queue moves the email out of the fragile, synchronous request path and into durable storage, where it can be retried independently of whether the user's HTTP request is still alive.

What does a queue actually buy me?

Four concrete benefits:

• Survives application restarts. A queued email persisted to a database or broker is still there after a deploy, crash, or scale-down. An in-flight fetch is not.
• Automatic retry handling. The queue worker owns the backoff loop, so your request handler stays simple and fast.
• Rate limit management. A queue is a natural throttle. You control the drain rate, so you never blow past the provider's limits during a burst.
• Audit trail. Every email has a row with a status and timestamps, which is invaluable for debugging "did the user get the email?" questions and for compliance.

There is a fifth, subtler benefit: a queue decouples your latency from the provider's. If the provider's API slows from 80ms to 4s during an incident, an inline send drags every user request down with it. A queued send returns to the user instantly and absorbs the provider's slowness in the background.

What is the simplest queue pattern?

A small state machine on top of one database table:

1. Write email to queue/database with "pending" status.
2. Process queue, attempt send.
3. On success: mark "sent", store message ID.
4. On retryable failure: increment retry count, schedule retry.
5. On permanent failure: mark "failed", alert.

This state machine, pending → sent | failed with a retry counter, is small enough to implement on top of a single database table, yet it gives you durability, observability, and a dead-letter path (rows stuck in failed) for free. Pair it with the idempotency keys above: persist the key with the queued job so that replays after the provider's key-retention window are handled deliberately, not by accident.

A copy-paste-ready outbox schema

The most robust pattern for a service that already has a relational database is the transactional outbox: you write the email row in the same database transaction as the business change that triggers it. The email cannot be lost (it commits atomically with the order) and cannot be sent for a rolled-back order.

CREATE TABLE email_outbox (
  id              BIGSERIAL PRIMARY KEY,
  idempotency_key TEXT        NOT NULL UNIQUE,
  recipient       TEXT        NOT NULL,
  template        TEXT        NOT NULL,
  payload         JSONB       NOT NULL,
  status          TEXT        NOT NULL DEFAULT 'pending'
                  CHECK (status IN ('pending','sending','sent','failed')),
  attempts        INT         NOT NULL DEFAULT 0,
  max_attempts    INT         NOT NULL DEFAULT 8,
  next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  provider_msg_id TEXT,
  last_error      TEXT,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Workers claim due jobs without stepping on each other.
CREATE INDEX idx_email_outbox_due
  ON email_outbox (next_attempt_at)
  WHERE status IN ('pending','failed') AND attempts < max_attempts;

A worker claims work atomically with FOR UPDATE SKIP LOCKED, so many workers can drain the table concurrently without double-sending:

// One claim-and-send cycle. Run this in a loop across N workers.
async function drainOnce(db, emailClient): Promise<boolean> {
  const job = await db.tx(async (t) => {
    const rows = await t.query(`
      SELECT * FROM email_outbox
      WHERE status IN ('pending','failed')
        AND attempts < max_attempts
        AND next_attempt_at <= now()
      ORDER BY next_attempt_at
      FOR UPDATE SKIP LOCKED
      LIMIT 1
    `);
    if (rows.length === 0) return null;
    const r = rows[0];
    await t.query(`UPDATE email_outbox SET status='sending', updated_at=now() WHERE id=$1`, [r.id]);
    return r;
  });

  if (!job) return false; // nothing due

  try {
    const result = await emailClient.send(job.payload, {
      headers: { 'Idempotency-Key': job.idempotency_key },
    });
    await db.query(
      `UPDATE email_outbox SET status='sent', provider_msg_id=$2, updated_at=now() WHERE id=$1`,
      [job.id, result.id],
    );
  } catch (error) {
    const attempts = job.attempts + 1;
    const giveUp = !isRetryable(error) || attempts >= job.max_attempts;
    // Queue-budget backoff: minutes, not milliseconds. Full jitter.
    const backoffMs = Math.min(2 ** attempts * 60_000, 2 * 60 * 60_000) * Math.random();
    await db.query(
      `UPDATE email_outbox
         SET status=$2, attempts=$3, last_error=$4,
             next_attempt_at = now() + ($5 || ' milliseconds')::interval,
             updated_at=now()
       WHERE id=$1`,
      [job.id, giveUp ? 'failed' : 'pending', attempts, String(error), Math.round(backoffMs)],
    );
    if (giveUp) await alertOnDeadLetter(job, error);
  }
  return true;
}

The idempotency key is the safety net for the one race this design cannot fully avoid: a worker marks a row sending, sends successfully, then crashes before writing sent. On recovery the row looks stuck and gets retried, but because the same key is replayed, the provider deduplicates and no second email goes out. The crash becomes harmless, which is the entire point.

How do I handle the dead-letter queue?

A job that exhausts max_attempts is in a dead-letter state (status='failed', attempts >= max_attempts). This is not a place messages go to die silently; it is a work queue for humans:

• Alert on entry, not on a dashboard nobody watches. A password reset reaching the dead-letter queue is an incident.
• Make replay a one-liner. Reset status='pending', attempts=0, next_attempt_at=now() once the underlying cause (provider outage, bad config) is fixed. Because the idempotency key is preserved, replay is safe if it happens within the provider's key window; beyond that window, your own outbox UNIQUE(idempotency_key) plus a status='sent' guard prevents a true duplicate.
• Distinguish poison messages from outage casualties. A message that fails with a permanent 4xx (bad address, oversized payload) should go straight to failed after one attempt, not burn eight retries. Only transient errors should consume the full retry budget.

Which emails actually need a queue?

Not every email needs one. A best-effort "we miss you" re-engagement email can send inline. But anything the user is waiting on (login codes, receipts, confirmations) or anything legally required benefits from the durability a queue provides. A simple rule: if a missed send would create a support ticket, a security gap, or a compliance problem, queue it.

A practical refinement is two queues with different drain rates: a low-latency "fast lane" for transactional, user-blocking emails, and a throttled "bulk lane" for marketing. This keeps a 200,000-recipient campaign from starving a user's password reset behind it, a real failure mode when both share one queue. The split also maps cleanly onto the transactional-vs-marketing distinction in Email Types and the separate-subdomain reputation advice in Deliverability.

Queuing checklist

• Critical emails are written to durable storage before the send is attempted (ideally a transactional outbox committing with the business change).
• Each job carries its idempotency key and retry count.
• Workers claim jobs with FOR UPDATE SKIP LOCKED (or equivalent) so concurrent drain is safe.
• Successful sends record the provider message ID.
• Permanently failed jobs land in a visible "failed" state and trigger an alert.
• Poison (permanent 4xx) messages dead-letter after one attempt, not the full retry budget.
• Transactional and bulk traffic use separate lanes / drain rates.
• The drain rate respects provider rate limits.

How do I stay within provider rate limits and warm up sending?

A queue is only a throttle if you actually throttle it. Providers enforce per-second, per-minute, and daily send limits, and exceeding them produces 429s that, if retried naively, snowball.

How do I shape outbound traffic to the provider's limit?

Drain the queue with a token-bucket limiter sized to the provider's documented rate. A token bucket allows short bursts up to the bucket size while holding the long-run average at the refill rate:

// Token bucket: `capacity` burst, `refillPerSec` steady-state rate.
class TokenBucket {
  private tokens: number;
  private last = Date.now();
  constructor(private capacity: number, private refillPerSec: number) {
    this.tokens = capacity;
  }
  tryRemove(): boolean {
    const now = Date.now();
    this.tokens = Math.min(
      this.capacity,
      this.tokens + ((now - this.last) / 1000) * this.refillPerSec,
    );
    this.last = now;
    if (this.tokens >= 1) { this.tokens -= 1; return true; }
    return false;
  }
}

Size the bucket conservatively below the provider's hard limit so transient bursts have headroom before you hit 429. When you do hit a 429, treat it as a signal to lower the drain rate temporarily (additive-increase / multiplicative-decrease), not just to retry the one message.

Why does ramping volume matter for a new domain or IP?

A brand-new sending domain or dedicated IP has no reputation. Sending 500,000 emails on day one from a cold IP looks exactly like a compromised host to receiving mailbox providers, and a large fraction will be deferred or spam-foldered. Warm-up means ramping volume gradually over days to weeks so reputation builds with engaged recipients first. This is a deliverability concern that constrains your sending architecture: your queue's drain rate must be capable of capping daily volume per sending identity during warm-up, not just throttling per-second bursts. The full warm-up curve and reputation mechanics live in Deliverability; the architectural takeaway here is that your rate limiter needs a daily ceiling, not only a per-second one.

What about authentication, unsubscribe, and bulk-sender rules?

Sending reliability is wasted effort if the provider accepts your message but mailbox providers reject it for failing authentication or bulk-sender policy. These are not "deliverability extras", since February 2024 they are hard preconditions for sending at any volume to the major mailbox providers, so they belong in any honest discussion of whether a send will actually succeed.

What do the Gmail, Yahoo, and Microsoft bulk-sender rules require?

Effective February 2024, Gmail and Yahoo (with Microsoft following the same direction for Outlook.com / Hotmail) enforce requirements on bulk senders (commonly framed around 5,000+ messages per day to their users, though the authentication baseline applies to everyone):

1. Authenticate your mail with SPF and DKIM, with a DMARC policy published (at minimum p=none).
2. Keep the spam-complaint rate below 0.3% as measured in postmaster tooling, ideally well under 0.1%. Above 0.3% you will see throttling and spam-foldering.
3. Support one-click unsubscribe on marketing/bulk mail per RFC 8058, and honor unsubscribe requests within two days.
4. Send from authenticated, aligned domains, the From domain must align with your DKIM/SPF for DMARC to pass.

These map directly onto reliability: a message that fails authentication is effectively an unreliable send, because it will be silently filtered regardless of how cleanly your API call succeeded.

What are the email authentication standards, by RFC?

Copy-paste-ready DNS records

These are illustrative records for a domain example.com sending via a generic provider. Replace selectors, includes, and reporting addresses with your own.

; SPF (RFC 7208), TXT at the root. Authorize your provider's sending hosts.
; Keep to a single SPF record; stay under the 10 DNS-lookup limit.
example.com.            IN TXT "v=spf1 include:_spf.your-provider.example -all"

; DKIM (RFC 6376), public key at a provider-specific selector.
; The provider gives you the selector (here "s1") and the key.
s1._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQ...IDAQAB"

; DMARC (RFC 7489), start at p=none for monitoring, then move to quarantine/reject.
_dmarc.example.com.     IN TXT "v=DMARC1; p=none; rua=mailto:dmarc-reports@example.com; adkim=s; aspf=s; pct=100"

; MTA-STS (RFC 8461), declares you want enforced TLS for inbound mail.
_mta-sts.example.com.   IN TXT "v=STSv1; id=20260101T000000Z"

; TLS-RPT (RFC 8460), where TLS failure reports are sent.
_smtp._tls.example.com. IN TXT "v=TLSRPTv1; rua=mailto:tls-reports@example.com"

; BIMI, published only after DMARC is at enforcement (quarantine/reject).
default._bimi.example.com. IN TXT "v=BIMI1; l=https://example.com/logo.svg; a=https://example.com/vmc.pem"

Roll out DMARC deliberately: publish p=none first and read the aggregate (rua) reports until SPF and DKIM align for all legitimate streams, then advance p=none → p=quarantine → p=reject, optionally ramping with pct=. Moving straight to p=reject before your streams are aligned will reject your own legitimate mail.

Verifying records with dig

# SPF, confirm exactly one v=spf1 TXT record exists.
dig +short TXT example.com | grep spf1

# DKIM, check the public key at your provider's selector.
dig +short TXT s1._domainkey.example.com

# DMARC, confirm policy and reporting addresses.
dig +short TXT _dmarc.example.com

# MTA-STS policy file is fetched over HTTPS, but the TXT advertises it:
dig +short TXT _mta-sts.example.com
curl -s https://mta-sts.example.com/.well-known/mta-sts.txt

# TLS-RPT reporting endpoint.
dig +short TXT _smtp._tls.example.com

How do I implement one-click unsubscribe (RFC 8058)?

One-click unsubscribe requires both the List-Unsubscribe header (with at least one https: URL, a mailto: alone does not satisfy the one-click requirement) and the List-Unsubscribe-Post header. When both are present, mailbox providers render a built-in "unsubscribe" control and POST to your URL when the user clicks it, with no further interaction.

List-Unsubscribe: <https://example.com/u/unsub?token=OPAQUE_TOKEN>, <mailto:unsubscribe@example.com?subject=unsub>
List-Unsubscribe-Post: List-Unsubscribe=One-Click

// The endpoint must accept a POST (not a GET) and unsubscribe immediately,
// without requiring a login or a confirmation page. RFC 8058.
app.post('/u/unsub', async (req, res) => {
  // Mailbox providers send: List-Unsubscribe=One-Click in the body.
  const token = req.query.token as string;
  const sub = verifyUnsubToken(token); // opaque, signed, identifies recipient+list
  if (!sub) return res.sendStatus(400);

  await suppressRecipient(sub.recipient, { reason: 'one-click-unsubscribe', list: sub.list });
  return res.sendStatus(200); // honor within 2 days; immediate is best
});

Two rules that catch people out: the token must be opaque and signed so an attacker cannot unsubscribe arbitrary addresses by guessing IDs, and the handler must be a true POST that acts immediately, a GET-based "click here, then confirm" flow does not satisfy RFC 8058 and will not earn the native unsubscribe button. Unsubscribe and suppression mechanics are covered further in Compliance and List Management.

What timeout should I set on an email API call?

Set a timeout of 10 to 30 seconds, and always clean it up. A request with no timeout can hang indefinitely if the provider is slow or the network stalls, tying up a connection, a thread, or a serverless function's billed duration, and (worse) blocking the user behind a spinner with no resolution.

const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 10000);

try {
  await emailClient.send(emailData, { signal: controller.signal });
} finally {
  clearTimeout(timeout);
}

Two details make this correct:

• The AbortController gives you a handle to cancel the in-flight request. After 10 seconds (10000 ms), controller.abort() fires and the send rejects rather than hanging forever.
• The finally block clears the timer in all cases: success, failure, or abort. Forgetting this leaks timers and, in some runtimes, keeps the process alive longer than necessary.

Recommended range: 10 to 30 seconds for email API calls. Below about 10s you risk aborting requests that would have succeeded on a slightly slow connection; above about 30s you keep resources tied up long past the point where retrying would have been faster.

Which timeouts actually matter, and why one number isn't enough?

A single overall timeout is a blunt instrument. A robust HTTP client distinguishes several phases, because a stall in each one means something different:

The crucial distinction is between a connect timeout and a request (read) timeout. A connection that never establishes (ECONNREFUSED, connect timeout) almost certainly did not send the email, so it is unambiguously safe to retry. A request that times out after the body was transmitted is the dangerous case, the email may or may not have been accepted, and is exactly why the idempotency key is mandatory before you enable retries. Where your HTTP client supports it, set a short connect timeout and a separate, longer read timeout rather than one combined value.

How do timeouts interact with retries?

They compose cleanly. A timeout produces a retryable error (ETIMEDOUT in the isRetryable check above), so a request that hangs is aborted at 10s, then re-attempted with backoff, which is exactly the behavior you want. The timeout bounds each individual attempt, and the retry loop decides whether to try again.

One subtlety: make sure your total budget (attempts × timeout + backoff delays) stays under any enclosing deadline. If you are inside an HTTP request handler with a 30s gateway timeout, three 10s attempts plus backoff will blow past it, the gateway will kill the connection, and the user gets an error even though your retries might still succeed. When the in-process budget can't fit under the enclosing deadline, do one fast attempt and hand the rest to the queue.

How do all of these fit together in one send?

The four pillars reinforce one another, and a production sender uses all of them at once:

1. Idempotency makes retries safe. Without it, retry logic would create duplicates.
2. Retry logic makes transient failures recover. Without it, every network blip becomes a lost email.
3. Error handling makes each failure class get the right treatment: fix, retry, suppress, or escalate.
4. Queuing makes the whole thing durable, surviving crashes and decoupling sending from the request path.
5. Timeouts keep individual attempts bounded, feeding cleanly into the retry loop.
6. Rate limiting and authentication make the accepted message actually deliverable, not just accepted.

A single critical send therefore looks like this: write a pending job to the outbox in the same transaction as the business change, a worker claims it with SKIP LOCKED and attempts the send with an idempotency key and a 10s timeout, on 5xx, 429, 503, or timeout it backs off (queue-budget, full jitter) and retries, on a 550-class hard bounce it suppresses the recipient, on success it stores the message ID, and on exhausting the retry budget it dead-letters and alerts. The end-to-end guarantee is exactly-once delivery with automatic recovery and no silent failures.

A sequence walkthrough of the ambiguous-failure case

To see why the pieces interlock, trace the worst case, a timeout after the request was sent:

1. Worker claims outbox row 42 (key K), marks it sending, sets a 10s timeout.
2. The provider receives the request, accepts the message, but the response is lost; the worker's timeout fires (ETIMEDOUT).
3. isRetryable returns true; the worker schedules a queue-budget retry and the row goes back to pending.
4. Later the worker re-claims row 42 and re-sends with the same key K.
5. The provider recognizes K, returns the original message ID without sending again.
6. The worker writes sent + the message ID. Exactly one email reached the recipient.

Remove any single pillar and this breaks: without the timeout the worker hangs forever; without the retry the email is lost; without idempotency step 5 sends a duplicate; without the durable queue a crash between steps 2 and 4 loses the message entirely.

Common mistakes

A consolidated list of the failures that recur most often in production email systems:

• Retrying without an idempotency key. The single most common way to spam users. Any retry path needs a key.
• Generating the idempotency key inside the retry loop. A fresh key per attempt is identical to having no key.
• Treating an API 200 as "delivered." It means "accepted by the provider." Delivery is observed via webhooks (Webhooks & Events).
• No jitter on backoff. Synchronized retries after an outage create a thundering herd that re-downs the provider.
• Ignoring Retry-After. You back off less than the server asked, get rate-limited harder, and look like a bad actor.
• Retrying hard bounces. Repeatedly sending to a 550/invalid address shreds sender reputation; suppress instead.
• Sending inline on the request path. A slow provider drags down every user request and a crash loses the email.
• Sharing one queue for marketing and transactional. A campaign starves time-critical password resets.
• Logging full payloads. Reset links and tokens in your log aggregator are a credential leak.
• No daily volume ceiling on a new domain/IP. Cold-start blasting tanks deliverability before reputation can build.
• DMARC straight to p=reject before alignment. You reject your own legitimate mail.
• GET-based unsubscribe. Fails RFC 8058; you lose the native one-click button and risk bulk-sender penalties.

Troubleshooting guide

End-to-end reliability checklist

• Idempotency keys on every send, derived from stable business events.
• Retries only on idempotent paths; classified, exponential, jittered, and capped.
• Retry-After honored on 429/503 (both seconds and HTTP-date forms).
• Every error branch logs, queues, suppresses, or alerts. Nothing is silent.
• Hard bounces (550-class) feed suppression, not retries.
• Critical emails go through a durable queue (ideally a transactional outbox) with a visible failed state and dead-letter alerting.
• Transactional and bulk traffic use separate lanes.
• Drain rate respects provider per-second and daily limits; new domains/IPs are warmed up.
• Timeouts (10 to 30s) on every API call, with cleanup in finally and separate connect/read timeouts.
• Total retry budget fits under any enclosing request/gateway deadline.
• Provider message IDs stored to correlate with webhook events.
• SPF (RFC 7208), DKIM (RFC 6376), and DMARC (RFC 7489) published and aligned; complaint rate under 0.3%.
• One-click unsubscribe (RFC 8058) implemented as a signed POST endpoint.

What should I read next?

• Webhooks & Events: process delivery confirmations and failures.
• List Management: handle bounces and suppress invalid addresses.
• Deliverability: sender reputation, authentication, warm-up, and why throttling matters.
• Transactional Emails: what these critical emails should contain.
• Compliance: unsubscribe obligations and data-minimization in logs.
• Index