Queue, webhook, and sending reliability incidents

First 15 minutes

1. Record the ready backlog and the oldest unprocessed message age, using SQS ApproximateNumberOfMessagesVisible and ApproximateAgeOfOldestMessage or equivalent metrics.
2. Compare in-flight work, consumer errors, and throughput before adding workers or starting a drain.

Now

• Stop blind replay, isolate stored 5yz jobs for intervention, and keep only classified transient jobs eligible for controlled retry.

Next 24 hours

• When SES returned the documented quota or maximum-rate error, wait up to ten minutes before retrying API sends and restart at a controlled rate.

Next 7 days

• Tune consumer capacity from ready, in-flight, error, and throughput evidence instead of backlog count alone.

Technical checks

Verification criteria

• Ready backlog and oldest-message age both trend down from the incident baseline while completed throughput continues.
• In-flight work no longer indicates stuck consumers when compared with worker errors and throughput.

Escalation criteria

• Escalate to engineering when oldest-message age keeps growing despite healthy throughput, and to SES support when the documented throttling condition persists after provider-directed retry handling.

Prevention

• Alert on ready backlog, oldest-message age, and in-flight work together, require idempotent logical sends, and preserve response classification for every queued job.

Business impact

• A growing visible backlog or oldest-message age means customer-facing email is waiting longer even when the queue count appears stable.

Provider notes

• Amazon SQS backlog and age metrics are approximate and should be interpreted with lifecycle, in-flight, error, and throughput evidence.
• Amazon SES recommends waiting up to ten minutes before retrying API sends only for its documented daily-quota or maximum-rate throttling errors.

Open questions

• The queue technology, queue depth, oldest-message age, in-flight count, retry schedule, and worker health were not provided.
• The root cause is unresolved until downstream responses, quota events, worker errors, and backlog age are correlated.
• It is not known whether consumers are idempotent or whether draining the backlog could duplicate customer-visible email.

First 15 minutes

1. Resolve public MX for the exact dedicated receiving hostname and confirm that the priority-10 target is mx.sendgrid.net.
2. Test the configured destination URL from the public internet and capture the final HTTP status without relying on redirects.

Now

• Correct the dedicated MX target or restore public reachability, and remove any redirect from the configured destination path.

Next 24 hours

• Fix multipart handling and, when enabled, verify Twilio signature and timestamp headers against the raw request body before parsing.

Next 7 days

• Make durable acceptance precede the 2xx response and alert on 5xx retries early enough to act before the documented three-day drop point.

Technical checks

Verification criteria

• Public DNS shows the dedicated hostname's priority-10 MX target as mx.sendgrid.net, and the configured URL is reachable from outside the private network.
• A controlled multipart event passes any configured raw-body signature check, is durably accepted, and receives a direct valid 2xx response.

Escalation criteria

• Escalate to Twilio SendGrid when 5xx retries persist and undelivered posts are approaching the documented three-day drop window.

Prevention

• Continuously test dedicated MX routing, public endpoint reachability, multipart parsing, raw-body signature verification when enabled, and durable acceptance before 2xx.

Business impact

• Inbound events that remain undelivered after documented 5xx retries can be dropped after three days without advance notification, leaving dependent business processing incomplete.

Provider notes

• SendGrid does not follow HTTP redirects, and a valid 2xx removes the item from its retry queue.
• The accepted evidence supports retries for 5xx responses and a three-day drop point; behavior outside that documented server-error scope remains unresolved.

Open questions

• The receiving hostname, public MX answer, Inbound Parse configuration, destination URL, and endpoint request logs were not provided.
• The current failure point is unresolved until DNS delivery, HTTP status, redirect behavior, multipart parsing, and signature verification are tested end to end.
• Twilio's Help Center says any non-2xx response or DNS failure is retried for 72 hours, while the developer page describes 5xx retries specifically; confirm current 3xx and 4xx behavior with Twilio before relying on it.

First 15 minutes

1. Confirm that the affected Triggered Send definition is in Running or Active status.
2. If it is Running with a non-empty queue, read the Alert Manager reason and identify the error before restarting the definition.
3. Treat each initial asynchronous success as structural validation only, not proof of final processing.

Now

• For an inactive definition, start or restart it and observe the eligible queued messages.
• For a Running definition in error, resolve the Alert Manager cause before applying the Stop, Publish, Start sequence.

Next 24 hours

• Reconcile callback results, retained request and response data, RecipientSendId records, and the NotSent extract for the affected period.

Next 7 days

• Build final callback handling and retention of complete request, response, and RecipientSendId evidence into the Triggered Send integration.

Technical checks

Verification criteria

• The definition remains Running or Active while controlled asynchronous tests reach a final callback result.
• Final callbacks account for initial asynchronous successes without unexplained later failures.
• RecipientSendId and NotSent checks account for the affected subscribers.

Escalation criteria

• Escalate when a Running definition still has a queue after the Alert Manager cause is resolved and the documented restart sequence is completed.
• Provide Salesforce support with callback results, complete requests and responses, RecipientSendId values, and NotSent evidence when those records do not explain the delay.

Prevention

• Treat the final asynchronous callback as the processing result instead of relying on the initial successful response.
• Retain RecipientSendId values and review NotSent data so excluded subscribers remain diagnosable.

Business impact

• Paused or Inactive definitions can leave intended triggered messages queued instead of sent.
• Subscribers queued longer than 72 hours can move to error when the definition restarts.
• An asynchronous request can appear initially successful even when later processing fails.

Provider notes

• Salesforce applies a three-day expiration policy to subscribers queued for the covered Triggered Send types.

Open questions

• The exact incident cause requires the tenant's triggered-send status, queue state, callback results, NotSent records, and request/response logs.

First 15 minutes

1. Inspect the endpoint's own logs for 5xx responses and repeated delivery symptoms because SendGrid does not provide a dashboard failure alert.
2. Restore the endpoint so it returns a valid 2xx when it accepts the delivery, before the documented three-day drop point.

Now

• Restore the endpoint and return a valid 2xx only when it accepts the Inbound Parse delivery.

Next 24 hours

• Monitor 5xx responses and retry symptoms in endpoint logs while service remains restored.

Next 7 days

• Exercise endpoint recovery so the team can restore valid 2xx acceptance before the documented three-day drop point.

Technical checks

Verification criteria

• A controlled Inbound Parse POST receives a valid 2xx and is removed from SendGrid's retry queue.
• Endpoint monitoring shows accepted deliveries and no recurring 5xx retry symptoms after restoration.
• Local monitoring detects failed POSTs without relying on a SendGrid dashboard alert.

Escalation criteria

• Escalate to SendGrid when the endpoint is accepting valid 2xx responses but deliveries still retry or approach the documented three-day drop point, and include endpoint logs because no dashboard alert exists.

Prevention

• Continuously monitor endpoint 5xx responses and restore valid 2xx acceptance before the three-day drop point.

Business impact

• Inbound messages that remain undelivered after three days are dropped without prior notification.
• The original sender receives no bounce and operators receive no SendGrid dashboard alert for failed POSTs, creating a monitoring blind spot.

Provider notes

• The published automatic-retry behavior is specific to 5xx responses and does not establish a universal rule for 3xx, 4xx, or DNS failures.

Open questions

• Twilio does not publish the exact schedule of increasing retry intervals within the documented 72-hour retry window.

First 15 minutes

1. Recheck the official Postmark live status because the operational observation from 2026-07-17 can change.
2. Classify representative affected messages as Queued, Processed, or Delayed in Postmark Activity.

Now

• Confirm and correct any account or Message Stream pause before submitting more mail to that stream.

Next 24 hours

• Separate recipient-server Processed or Delayed cases from Postmark Queued cases so each is handled by the responsible party.

Next 7 days

• Document the Activity decision path for Queued versus Processed or Delayed messages.

Technical checks

Verification criteria

• The current live status and present unplanned-incident notices are consistent with any affected messages that remain Queued in Activity.
• Affected messages no longer remain Queued after any account, Message Stream, or provider condition is resolved.

Escalation criteria

• If current status and Activity do not explain the delay, contact Postmark support with a link to an affected Activity message after checking local submission logs.

Prevention

• Monitor the official current status and present unplanned-incident notices.
• Retain Activity links and local submission evidence so unexplained cases can be escalated with a specific affected message.

Business impact

• Queued messages have been accepted by Postmark but remain unsent, which delays the intended communication.
• Processed or Delayed messages can be held or temporarily refused by the recipient server rather than by a Postmark-wide incident.

Provider notes

• For Delayed messages, Postmark retries most domains every 10 minutes for up to 12 hours, but some domains use different rates and durations.

Open questions

• The user-specific cause and whether a delay remains current after verification are unknown without Postmark Activity links, submission logs, and current status evidence.