Transactional Email Outages

First 15 minutes

1. Inspect Amazon SES DeliveryDelay details, including recipient, enhanced status, diagnostic, delay type, event time, and expiration time when publishing is enabled.
2. Measure request-to-provider, provider-to-delivery, and delivery-to-click latency against the token lifetime.

Now

• Issue a newly generated token through a rate-limited resend path when the delayed message is already expired.

Next 24 hours

• Keep prior reset tokens invalid after use or replacement according to the application's outstanding-token model.

Next 7 days

• Set a reset-specific retry and alert budget shorter than the configured token lifetime.

Technical checks

Verification criteria

• Measured request-to-provider, provider-to-delivery, and delivery-to-click latency remain within the configured token lifetime for the tested flow.

Escalation criteria

• Escalate to the ESP when DeliveryDelay diagnostics show the provider queue approaching expiration before the reset-specific budget.

Prevention

• Use single-use reset tokens with a lifetime selected for the application's threat model and recovery experience.
• Monitor each latency segment against token expiry rather than relying on provider acceptance alone.
• Keep the reset retry and alert budget shorter than the token lifetime.

Business impact

• Users can receive unusable recovery links because general SMTP retry windows can exceed a single-use token's appropriate lifetime.

Provider notes

• Amazon SES DeliveryDelay events include the delay diagnostic and expiration time only when that event publishing is enabled.

Open questions

• The configured token TTL, observed latency distribution, retry schedule, and delay stage are not provided.
• ESP and recipient MTAs use different retry schedules and delay events, so the RFC and SES examples do not establish the actual queue behavior.

First 15 minutes

1. Check whether the order event came through an SQS standard queue that can return the same message copy more than once.
2. Compare the idempotency keys for duplicate confirmations and confirm one stable event-type and order-ID key was reused.
3. Check whether Resend returned 409 for a changed payload or an identical keyed request still in progress.

Now

• Reuse the same Resend idempotency key and payload for every retry of one order-confirmation request.

Next 24 hours

• Derive a stable key from the event type and order ID so one order event maps to one logical send.

Next 7 days

• Keep concurrent retries on the same key and retry the in-progress 409 case later with the same request.

Technical checks

Verification criteria

• Repeat the same keyed Resend request within the documented 24-hour retention window and confirm it is processed once.
• Confirm a key reused with a different payload returns 409 rather than creating another logical send.
• Confirm an identical keyed request that was in progress can be retried later with the same request.

Escalation criteria

• Escalate the duplicate-protection design when it must cover a period longer than Resend's documented 24-hour idempotency-key retention.
• Escalate to the ESP when observed 409 behavior differs from the documented changed-payload or in-progress cases.

Prevention

• Make standard-queue consumers idempotent so message redelivery cannot create duplicate effects.
• Use the same provider idempotency key on every retry.
• Derive one stable event-type and entity-ID key for each logical order-confirmation send.

Business impact

• Without the same Resend idempotency key on every retry, repeated send requests are not protected by the documented process-once behavior.

Provider notes

• Resend retains email idempotency keys for 24 hours.
• Resend returns 409 when a key is reused with a different payload or while an identical keyed request is still in progress.

Open questions

• The duplicate's origin could be repeated order events, queue redelivery, concurrent workers, an HTTP retry, or distinct idempotency keys; logs must distinguish them.
• Email API idempotency support, key retention and conflict semantics vary by ESP; Resend's documented 24-hour behavior is not universal.
• The application's current durable send ledger and uniqueness constraints are unknown.

First 15 minutes

1. Trigger the registration and password-reset templates separately and label every test by flow.
2. For each trigger, distinguish successful API acceptance from recipient-server handoff by comparing sent and delivered events.
3. Inspect delivery-delayed and failed event details before assigning a cause.

Now

• Correct the invalid-recipient, credential, domain-verification, or quota issue identified in the failed-event detail.

Next 24 hours

• Review whether the same failed-event cause appears in both authentication flows or only one.

Next 7 days

• Document the response for each failed-event cause observed in the deployed provider.

Technical checks

Verification criteria

• Separate controlled signup and password-reset tests reach the provider's delivered state rather than stopping at sent.
• The two test flows remain distinguishable in event records, so a recurrence can be isolated by flow.

Escalation criteria

• Escalate to the email provider when delivery-delayed events persist without a usable temporary-cause detail.
• Escalate to the owning engineering team when failed events identify credentials, domain verification, or quota as the cause.

Prevention

• Keep separate controlled signup and password-reset tests so both authentication paths can be checked independently.

Business impact

• The incident may disrupt sign-up confirmation, password reset, or both distinct authentication flows.

Provider notes

• The sent, delivered, delivery-delayed, and failed names in this runbook are Resend event semantics; map equivalent states for another provider.

Open questions

• The affected platform, provider, environment, and message IDs are not supplied; exact root cause cannot be determined from the question alone.
• Whether registration, password reset, or both are affected must be measured with separate controlled triggers and event traces.
• Provider event names and retry behavior vary; map the documented Resend states to the deployed provider.

First 15 minutes

1. Identify whether the project uses the Resend connector or Lovable Cloud managed email.
2. For a Resend connector, check the API key, sender or domain setup, project linkage, and rejected or failed activity.
3. For Lovable Cloud, confirm that the custom domain is Verified and email is enabled for the project.

Now

• Restore the correct Resend API key, sender or domain setup, and project linkage when the connector checks identify a mismatch.

Next 24 hours

• Restore Lovable Cloud custom-domain sending only after the domain is Verified and email is enabled for the project.

Next 7 days

• Document the separate connector-linkage and Cloud-domain controls so future incidents are routed to the correct path.

Technical checks

Verification criteria

• A new connector test appears in the intended Resend account without rejected or failed activity.
• A Lovable Cloud test is sent only after the custom domain shows Verified and project email is enabled.

Escalation criteria

• Escalate a reproducible platform bug or missing workaround to Lovable Support, and use the Lovable status page for platform-wide incident updates.

Prevention

• Record which projects are linked to the workspace's Resend connector and treat unlinking as removal of email access.
• Monitor Lovable Cloud domain verification and project email enablement as separate controls.

Business impact

• An unlinked Resend connector removes the project's access to that email connection.
• Lovable Cloud custom-domain mail requires both a Verified domain and email enabled for the project.

Provider notes

• Lovable Cloud managed email is documented at 100 messages per hour per workspace; this limit does not describe the separate Resend connector.

Open questions

• The question does not identify whether the app uses Lovable Cloud managed email or the separate Resend connector; their controls and logs differ.
• No project logs, connector state, domain status, HTTP error, quota state, or provider message ID was supplied, so the exact cause is unknown.
• A tenant-specific failure cannot be attributed to a Lovable-wide incident without contemporaneous status evidence.

First 15 minutes

1. Correlate API acceptance, sent, delivery-delayed, and delivered timestamps for controlled OTP requests.
2. Compare the measured handoff time with the project's configured OTP expiry rather than assuming the hosted default.

Now

• Use the Supabase Send Email Auth Hook to queue authentication email when traffic spikes are contributing to delay.

Next 24 hours

• Configure the Send Email Auth Hook queue to smooth observed authentication-email spikes.

Next 7 days

• Evaluate routing the Send Email Auth Hook through multiple sending services for reliability.

Technical checks

Verification criteria

• Controlled requests reach the delivered event before the project's configured OTP expiry.
• The measured sent-to-delivered interval remains visible for each controlled OTP request.

Escalation criteria

• Escalate to the provider when delivery-delayed events persist and their temporary-cause details do not explain recovery.
• Escalate to engineering when failed events identify recipient, credential, domain-verification, or quota errors.

Prevention

• Keep the configured request interval explicit rather than assuming Supabase's 60-second hosted default.
• Use a queued Send Email Auth Hook and, where appropriate, multiple sending services to smooth spikes and improve reliability.

Business impact

• An OTP that arrives after the configured expiry cannot be used within that validity window.

Provider notes

• The one-hour OTP expiry and 60-second request interval are Supabase hosted defaults; project configuration may differ.

Open questions

• The deployed auth provider, configured OTP lifetime, queue timestamps, provider events, recipient provider, and observed latency distribution are not supplied.
• The one-hour expiry and 60-second request window are Supabase defaults, not universal OTP requirements.
• It is unresolved whether delay occurs before provider acceptance, during provider delivery, or after recipient-server handoff.

First 15 minutes

1. Confirm whether the missing stream is login verification from noreply@salesforce.com rather than password reset from support@salesforce.com.
2. Request Salesforce Email Log Files to determine whether the verification message was sent or failed.
3. Check Login History and Identity Verification History for the challenge and destination Salesforce attempted.

Now

• When only login verification is filtered, trace and allowlist noreply@salesforce.com for that stream.

Next 24 hours

• Use Message-ID or sender tracing to confirm that the allowlist change affects the intended verification stream.

Next 7 days

• Document the distinct Salesforce senders and the five-per-hour verification-code request limit in the support runbook.

Technical checks

Verification criteria

• Salesforce Email Log Files show whether the controlled verification message was sent or failed.
• If the log shows recipient-server delivery, the recipient-side trace identifies the subsequent filter disposition.

Escalation criteria

• Escalate to the recipient mail administrator when Salesforce logs show delivery to the recipient server but the message is still filtered.
• Escalate the sender-specific filtering case when Message-ID tracing and allowlisting noreply@salesforce.com do not restore the login-verification stream.

Prevention

• Monitor the noreply@salesforce.com verification stream separately from the support@salesforce.com password-reset stream.
• Retain Salesforce email logs and challenge history so handoff can be distinguished from recipient filtering.

Business impact

• Filtering the noreply@salesforce.com login-verification stream can prevent users from receiving that verification path even when password-reset mail is unaffected.

Provider notes

• Salesforce documents noreply@salesforce.com for login verification, support@salesforce.com for password reset, and a five-code-request-per-hour limit for the described verification flow.

Open questions

• The Salesforce Email Log Files, Login History, Identity Verification History, Message-ID, recipient domain, and filter verdict are not supplied.
• Salesforce sender addresses and code limits are scoped to the documented flows and may change by product or verification method.
• Without the Salesforce handoff result, the incident cannot yet be assigned to Salesforce sending or the recipient mail system.

First 15 minutes

1. Match the order event to the provider sent event and then check for recipient-server delivered handoff.
2. Check whether the recipient is suppressed after a hard bounce or spam complaint.
3. Before any retry, determine whether a valid Idempotency-Key can prevent a duplicate request.

Now

• Identify and correct the original hard-bounce or complaint cause before changing a suppressed recipient.

Next 24 hours

• Remove a suppression only after confirming the address and fixing the underlying cause.

Next 7 days

• Review whether resolved recipients are being suppressed again, which indicates the original cause remains.

Technical checks

Verification criteria

• A controlled order confirmation reaches the provider's delivered state rather than stopping at sent.
• Provider test addresses reproduce delivered, bounced, complained, and suppressed outcomes without fake recipients.

Escalation criteria

• Escalate to the provider when a message has a sent event but no explainable delivered or terminal event.
• Escalate to deliverability when a suppression cause cannot be corrected without risking another hard bounce or complaint.

Prevention

• Use Resend Idempotency-Key on retryable order-confirmation requests within its 24-hour retention window.
• Test delivered, bounced, complained, and suppressed paths with documented provider test addresses.
• Require the original bounce or complaint cause to be fixed before suppression removal.

Business impact

• A missing one-to-one order confirmation disrupts the transactional message triggered by the customer's purchase action.

Provider notes

• Resend's sent and delivered states, regional suppressions, test addresses, and 24-hour idempotency window are provider-specific controls.

Open questions

• The order event ID, recipient address, provider message ID, API response, delivery events, and suppression state are not supplied.
• The exact provider is unknown; Resend and Postmark facts describe provider-specific state and classification examples.
• Recipient-server handoff would not prove inbox visibility or reading by the customer.

First 15 minutes

1. Confirm that the recipient address is pre-authorized as a member of the project's organization team when default SMTP is in use.
2. Check whether Supabase returned HTTP 429 and whether the built-in provider's project-wide limit applies.
3. Trace the request beyond API success to downstream sent and delivered events.

Now

• Decide whether the staging authentication path requires the reliability of custom SMTP rather than default SMTP.

Next 24 hours

• Configure custom SMTP when the authentication email path is production or otherwise mission-critical.

Next 7 days

• Keep mission-critical authentication email on the custom-SMTP path recommended by Supabase.

Technical checks

Verification criteria

• A controlled OTP request reaches the downstream delivered event rather than only the sent or API-success state.
• The mission-critical path uses the configured custom SMTP service rather than Supabase default SMTP.

Escalation criteria

• Escalate to the project owner when a mission-critical authentication path still depends on best-effort default SMTP.
• Escalate the email configuration when custom SMTP is required but not yet present for the critical flow.

Prevention

• Track the built-in provider's two-messages-per-hour project-wide limit wherever it remains enabled.
• Use custom SMTP for production and other mission-critical Supabase authentication email.

Business impact

• A staging flow that relies on best-effort default SMTP cannot use an API success response as assurance of downstream recipient-server handoff.

Provider notes

• Supabase default SMTP restricts recipients to authorized project-team addresses, is best-effort, and is intended only for non-production use.

Open questions

• The OTP API and email provider are not identified; Supabase and Resend behavior must not be assumed for another stack.
• The recipient's project-team authorization, current rate-limit response, SMTP configuration, and downstream delivery events are not supplied.
• An API success response alone cannot identify where the downstream email pipeline stopped.

First 15 minutes

1. Separate successful API requests from delivery, bounce, complaint, and delay events for the affected message stream.
2. Check whether each retry obeyed the deployed provider's idempotency contract; for Resend, confirm the same key and payload were reused within 24 hours.

Now

• Add provider-supported idempotency to repeated send requests.
• Deduplicate SendGrid webhook events by sg_event_id when present.

Next 24 hours

• Set the Amazon SES maximum delivery duration for time-sensitive mail to the application's validity period.

Next 7 days

• Separate one-to-one transactional messages from newsletter, announcement, and other bulk streams using the deployed provider's equivalent controls.

Technical checks

Verification criteria

• A test send produces a terminal delivery, bounce, complaint, or delay outcome distinct from API acceptance.
• Monitoring reports terminal delivery, bounce, complaint, and delay events for the transactional stream instead of counting API acceptance as customer delivery.

Escalation criteria

• Escalate to the ESP and engineering when accepted sends lack the provider's terminal delivery, bounce, complaint, or delay outcome.

Prevention

• Use idempotent send requests and deduplicate provider event callbacks.
• Keep transactional streams separate from broadcast mail and alert on terminal delivery, bounce, complaint, and delay outcomes.

Business impact

• Time-sensitive messages such as one-time-password email can arrive outside the application's useful validity period unless the delivery window is aligned.
• Treating API acceptance as customer delivery can hide bounce, complaint, delay, or missing terminal-delivery outcomes.

Provider notes

• SendGrid retries an Event Webhook POST after a non-2xx response at increasing intervals for up to 24 hours.

Open questions

• The deployed ESP, its idempotency contract, webhook delivery guarantees, suppression state, and failover capabilities are not available.
• Business SLOs, token validity windows, acceptable duplicate risk, and observed event gaps must be supplied by the SaaS owner.