Shopify Webhook Replay Safety: Raw-Body Verification, Delivery Identity, Fixtures, and Dead-Letter Recovery

Define a verifiable delivery contract first

Break one callback into provable states

Shopify webhook reliability is not the same as seeing a JSON object arrive. It is a chain of evidence from the network boundary to a committed business fact. Write down the topic, shop domain, API version, event time, delivery identity, raw bytes, signature result, durable-write result, and handler result. The receiver should capture the request and return a clear response quickly; business work should continue through a retryable queue. A named state prevents an accepted request from being mistaken for a completed order update.

Keep the receive time, an allow-listed set of headers, a body digest, a protected raw-body reference, parse errors, handler version, and terminal status. The digest links logs, the raw body allows HMAC recomputation and fixture replay, and a business key determines whether the order or inventory actually changed. A parsed object is only a derivative because whitespace, escaping, and numeric representation can differ. Test a duplicate event, a truncated body, and an unknown topic and verify that each reaches its intended state. Related context: Shopify API overview.

A common failure occurs when middleware parses JSON and the application signs a re-encoded object. Field order or whitespace then causes a valid request to fail. The recovery boundary is the signing input and record format; it does not relax the secret, topic, or shop checks. If raw bytes are gone, stop automatic replay and reconcile the current fact through the relevant Shopify API instead of writing from incomplete evidence.

Use a delivery card with states such as received, verified, queued, processed, and replayed. Received means bytes are stored; verified means the HMAC matches the shop secret; queued means a durable message exists; processed means the business transaction committed; replayed identifies a controlled retry. Allow only documented transitions and record the reason. This separates security, queue, and domain failures without depending on one person reading a log in the right order.

Choose the topic, version, and event identity

Make routing and uniqueness traceable together

Create a webhook subscription register with the topic, API version, callback address, shop domain, application environment, creation source, and owner. Shopify's webhook documentation describes subscriptions around event topics, so a URL alone is not a reliable route key. Treat X-Shopify-Shop-Domain, X-Shopify-API-Version, X-Shopify-Webhook-Id, X-Shopify-Event-Id, and X-Shopify-Triggered-At as routing and diagnostic fields. An unknown or incomplete envelope belongs in isolation.

One business action can produce more than one topic, and older subscriptions can carry a different version. The event key should distinguish shop, topic, webhook identity, and business object identity; an order number or product handle alone is unsafe. In import, reinstall, and multi-shop cases, validate tenant context before doing work. Keep topic-to-handler mapping in configuration. An unsupported topic should be recorded and safely acknowledged according to policy, never guessed into a similar action. Related context: Shopify Liquid theme development guide.

A failure case is a global deduplication table shared by several shops. Two shops can use the same business-number shape, so a valid event for the second shop is discarded as a duplicate. Recovery starts by rebuilding tenant-scoped key space, preserving skipped raw bodies, and comparing the current order state. A routing fallback sends an unknown topic to an inspection queue; it does not turn it into another topic or silence every webhook.

The register should also carry the last successful receipt, the last processing error, and signals for a removed or unhealthy subscription. Before a version change, fixtures should prove that header names are read without case sensitivity while required values still receive strict validation. Shopify's delivery structure reference lists useful headers and body boundaries; your logs must still prove what your receiver observed.

Persist the raw request body

Let signing, audit, and replay share one byte sequence

The receiver must establish that it has the raw body used for HMAC before JSON parsing, character conversion, framework middleware, or decompression changes it. Store the bytes in protected storage with length, digest, content type, and receive time; treat the parsed object as a derivative for domain work. Define states for an empty body, an oversized body, and invalid encoding. An exceptional input must not become a silent empty object.

The retention rule must balance privacy and recovery. Keep only the window needed for processing and investigation, mask sensitive fields in logs, and never rewrite the signed archive. Protect archive access with authorization and an audit trail. Name each archive through the shop, topic, webhook identity, and digest, then record its location in an index. When a fixture is read, compare its length and digest byte for byte so storage cannot introduce a newline or encoding change. Related context: Shopify store speed guide.

A typical failure is a framework body parser consuming the stream before the verifier reads it, leaving an empty string and making every request look forged. Recovery removes or reorders the consumer so the receiver copies the stream first. If the original bytes cannot be restored, reject replay and reconcile current facts. Do not accept arbitrary signatures or skip HMAC as a repair.

Durable storage can also slow down during a traffic burst or a full disk. A protected local buffer may hand bytes to a durable queue quickly, but the receiver should acknowledge only after it can prove that the bytes are readable and indexed. Buffer overflow needs an explicit signal and an operator path. Exercise a full disk, a write timeout, and a second read so loss cannot hide behind a successful HTTP response.

Verify the source with HMAC

Use a fixed order for secret, digest, and shop checks

The verifier should resolve the shop domain and its registered secret, compute HMAC-SHA256 over the raw body, and compare digests in constant time. It must not choose a secret from a shop field in the body or treat a query parameter as identity. Parse business data only after verification. Header names can be case-insensitive, while values, whitespace, encoding, and missing headers must follow an explicit contract.

During secret rotation, a short overlap can accept an old and a new secret, but every attempt should record the secret version without recording the secret. One request belongs to one normalized shop. Normalization should cover case, ports, and trailing dots without changing tenant identity. Fixtures using the same raw body with each secret prove that the transition cannot bind an unknown shop to another tenant. Related context: Shopify page builder comparison.

A failure case is an adapter that URL-decodes or base64-encodes the digest a second time to satisfy a proxy. Some requests then fail at random. Recovery restores a tested digest path, keeps constant-time comparison, and rejects signatures from query strings, cookies, or body fields. If a secret may be exposed, isolate the affected subscription and rebuild the shop-to-secret association instead of widening acceptance.

A signature failure response should be fast and should not reveal a secret, a computed digest, or an internal path. Classify failures as missing header, unknown shop, body mismatch, digest mismatch, clock anomaly, or duplicate receipt. Pair the classes with Shopify's webhook troubleshooting guidance, which emphasizes response and retry observation, rather than treating every 4xx or 5xx as the same alert.

Deduplicate with webhook identity

Separate repeated delivery from repeated business action

The deduplication key should at least contain the normalized shop domain, topic, and X-Shopify-Webhook-Id. If the identity header is absent, route the event to inspection; an empty string must never become a global key. Write receipt status, first-seen time, latest retry time, and a business-result reference with the deduplication row. A hit only proves that the delivery was seen; it does not prove that its previous business transaction committed.

Distinguish received-but-unprocessed, processed, and retryable failure. The transaction should record the business idempotency key and result before the queue message is acknowledged. If a downstream write times out, keep a retryable state and let the next attempt inspect the current fact. Reads, index updates, and notifications also need a defined repeated-run behavior even when they look harmless. Related context: Shopify Flow automation guide.

A dangerous implementation writes the deduplication marker first, then the process crashes before the business write. The next delivery sees the marker and never runs. Recovery scans rows without a result reference, sends their protected raw bodies to an inspection queue, and uses a transaction plus current resource state. Do not erase the whole table; that can make a real duplicate invoke an external action again.

A database uniqueness constraint, conditional write, or versioned key-value operation can protect concurrent consumers, but test two consumers receiving one identity at the same time. Retention and compaction should reflect Shopify's retry window, the business reconciliation window, and audit needs. Remove only old records with a complete result; keep failures and disputes traceable.

Build safe replay fixtures

Replay original events instead of similar JSON

A replay fixture contains the raw body, required headers, shop context, receive time, original HMAC, expected topic, expected business key, and handling result. Remove real secrets and use a dedicated test shop with a non-writing downstream. If real fields must remain, mask them irreversibly and store any mapping separately. Label each fixture as first handling, duplicate delivery, damaged body, unknown topic, or downstream timeout.

The replay tool should default to validation without external side effects. It verifies shape and signature, points the handler at an isolated database, and then reports state differences. Only an explicitly authorized recovery run may access real business APIs, and it must still use the original event identity and idempotency condition. Results include handler version, rule version, input digest, output digest, and failure class so two runs can be compared.

A failure case is a script that reserializes JSON and replaces the event time with now. It cannot reproduce a signing defect and might treat an old order as a new action. Recovery disables adapters with side effects, preserves the fixture and difference report, and queries current facts. Do not weaken signature or deduplication checks just to make a fixture pass.

Cover out-of-order, duplicate, delayed, and missing-optional-field inputs. Define an expected result for each: an old event waits for a version check or reconciliation, a duplicate returns its known result, a delayed event compares event time with current state, and a malformed envelope enters a named error. Fixtures are not eternal; review them against the current API version and handler contract and mark stale ones non-executable.

Set queue and dead-letter boundaries

Acknowledge only when the evidence is durable

Separate the receiver from domain work with a durable queue. A message should carry an archive reference, shop, topic, webhook identity, attempt count, first-receive time, and handler route rather than copying a large body into every layer. Document the acknowledgement state machine: acknowledge after a committed transaction, delay transient failures, dead-letter permanent data errors, and hold security failures for inspection.

A dead-letter queue is a recovery list with an index, owner, retention period, and reason, not a trash can. Mark each item with the failure class, latest exception, allowed recovery action, and whether another try is safe. Monitor age, growth, topic mix, and shop mix. Limit concurrency per tenant so one large shop cannot hide many small shops with persistent failures.

A classic failure acknowledges immediately after receiving the message; the process exits before storage commits, so the queue never tries again. Recovery uses the archive index and deduplication state to find messages without a business-result reference and replays them one by one. If the queue itself is damaged, rebuild messages from the durable archive index, not by guessing from the business table.

Dead-letter permissions should be narrower than ordinary consumption and should record who requeued an item, when, and why. Before replay, lock the shop and business key and check whether a human already corrected the object. A completed event should be resolved without another action. Record the retention rule, export shape, and deletion evidence so cleanup cannot remove audit-critical raw material.

Choose retry, backoff, and throttling by error class

Let the failure type decide when to try again

Not every error deserves an immediate retry. A network outage, downstream throttling, or storage timeout can usually wait; a signature mismatch, unknown shop, or invalid contract should be isolated. Record attempt start and end, response class, downstream reference, and next attempt time. Add a small random spread to backoff so one webhook burst does not wake together and overload a dependency again.

Shopify's troubleshooting guidance describes retries over a limited window and warns that a persistently failing subscription can be removed. The receiver should answer quickly and use its archive as recovery evidence. Do not promise an endless retry. Set an attempt ceiling, a dead-letter threshold, and a reconciliation condition. Throttle per shop so one tenant's outage remains visible and cannot consume every worker.

A failure case is retrying every 5xx immediately. The queue grows during downstream maintenance, and recovery later writes the same order concurrently. The safe boundary is to pause the affected handler, keep receipt and verification active, and resume in shop, topic, and event-time batches. Never erase attempt counts or disguise throttling as success.

A recovery console should preview the next events with shop, topic, event time, business key, and expected side effect. Validate without side effects first, then allow a small batch. If an event is stale or the current fact has changed, switch to reconciliation instead of blind replay. Produce a run summary and an unresolved list so a temporary outage cannot become unexplained state drift.

Handle ordering and event time

Respect state versions instead of arrival order

Webhook arrival order is not business order. Persist X-Shopify-Triggered-At, the business object's update version, processing time, and the last applied version. Before writing, compare versions or event times. For orders, inventory, and fulfillment, document which transitions are reversible and which require a current API read. A queue's FIFO order cannot safely overwrite a newer fact with an older event.

A safe pattern records every event and applies a conditional update only when the incoming version is not older than the stored version. When versions cannot be compared, reconcile the current object and rebuild the domain result. Keep timezone and original representation for time fields. Do not substitute receive time for event time or trust a client timestamp as the ordering authority.

A failure case sends a refund event first, then an older payment update marks the order paid again. Recovery locks the business key, pauses automatic overwrites, and rebuilds the legal path from the current order and archive. It may write an audit fact without calling a refund or notification a second time. Exercise out-of-order fixtures before reopening the queue.

Ordering tests should include old, new, duplicate, and incomplete events for one object, plus two shops with the same business-number shape. The result must explain whether an event applied, skipped, waited for reconciliation, or entered dead-letter. Store the version decision in logs; a final state alone cannot explain why a webhook did not change a storefront or an order record.

Monitor delivery health in layers

Locate source, queue, and domain failures separately

Use four monitoring layers: receive security, queue health, handling result, and business consistency. The receive layer shows signature failures, unknown shops, body size, and response latency. The queue layer shows oldest-message age, dead letters, and backlog by topic. The handling layer shows success, retry, conflict, and parse errors. The consistency layer compares sampled webhook facts with current API reads. Slice every layer by shop and topic so a global average cannot hide a local outage.

Correlate with irreversible digests, webhook identity, and business keys; do not log secrets, full access tokens, or unmasked sensitive data. A trace should connect the receipt log, archive index, queue message, database transaction, and downstream request. An alert should name the first action, such as checking raw-body storage, confirming downstream health, or pausing one topic, rather than merely saying webhook failed.

A failure case watches only HTTP 200. The receiver acknowledges quickly while its queue fails for hours, and business data never changes. Recovery begins with an archive-to-result sample and pauses automatic recovery until the gap is understood. If monitoring is incomplete, read durable logs and database state. Do not lower alert counts by classifying unknown and failed work as success.

Use fixed samples across frequent and rare topics, new shops, large bodies, and malformed bodies. Each exercise records an expected state. The health view should compare the subscription register with actual receipts so a removed subscription or bad callback is visible. Historical data supports trend and audit; it should not become a universal promise about business outcomes.

Recover data after an outage

Restore facts before restoring automation

When the receiver, queue, or downstream is unavailable, mark the start and end time, affected shops, and topics while protecting archives and logs. When service returns, do not push the entire backlog into business writes. Check signatures, duplicate state, current object versions, and downstream capacity first. Shopify guidance recommends importing objects from the outage window when webhook data is missing; reconciliation must use current API facts rather than message counts.

There are two recovery paths. Events with complete raw bodies can pass deduplication and version checks before replay. Events without provable identity should be found through an object query for the affected time window and placed in a difference list. Keep the source, query time, current state, proposed action, and human confirmation for each difference. Start with non-side-effect indexes and audit facts before orders, inventory, or notifications.

A failure case sends every historical event to a notification service after recovery. Customers receive duplicates and inventory may be reduced twice. The boundary is to pause side-effect handlers, run consistency queries, and review current facts. Never delete the archive or collapse a recovery result into an unexplained synced label.

Recovery is complete only when the time window was queried, missing objects have an explanation, duplicates have evidence, exceptions have owners, the queue is healthy, and samples agree with the API. Store a timeline, query parameters, and result summary. Change retention or retry settings only after the root cause is understood so several variables do not move at once.

Failure cases and the recovery boundary

Start with the smallest safe action

Case one: a body parser runs before HMAC and every product update with special escaping fails verification. Compare original content length, archive digest, proxy body, and computed digest. After confirming the cause, restore raw-stream access only and keep the shop and topic allow-lists. Case two: the deduplication key omits shop domain, so equal-shaped identifiers collide across shops. Rebuild tenant-scoped keys and reconcile skipped events one by one.

Case three: a consumer acknowledges before the database transaction and exits, while logs show success. Compare acknowledgement time, transaction reference, and archive index, then find messages without a result and run a side-effect-free replay. Case four: a throttled API is retried immediately and backlog spreads to every shop. Pause the handler, keep receipt and verification, and resume by tenant instead of changing signature rules or deleting the queue. The shared receiver reads raw bytes, verifies the source, normalizes headers, archives, deduplicates, and queues. A topic handler owns one domain's parsing, version check, and transaction. A handler should never reimplement HMAC or read the HTTP stream, because a local maintenance change could silently widen every entry point. Its interface receives a verified event envelope and a read-only archive reference. Each handler documents input fields, allowed state transitions, external calls, idempotency key, and failure classes. Liquid or storefront JavaScript should not carry webhook secrets; the application service and queue are the correct boundary. For a cross-domain action, write the local fact first and publish an internal message with a business key rather than making one webhook transaction update unrelated systems. A failure case is a generic handler that sees an unknown topic and calls a default order update. Recovery disables the default branch and sends the raw event to inspection; it does not guess missing fields. If the shared layer changes, fix it once and run every topic fixture, rather than applying one-off patches to handlers. Boundary checks cover topic allow-list, shop-secret selection, archive authorization, message envelope, handler version, and side-effect switch. Service accounts receive only the needed resource scope, and archive reading is authorized separately from recovery execution. The Shopify API integration guide is useful context, but server-side webhook evidence still comes from raw bytes and the official delivery contract. Whenever a topic, API version, callback, secret, queue, or handler changes, recheck the subscription register, official headers, raw-body retention, HMAC, deduplication, ordering, retry, dead-letter, and reconciliation paths. Use fixed fixtures to validate safe failures and then successful transactions, followed by a small sample of real objects. A single successful response is not proof of consistency, and an empty queue is not proof of correctness. The checklist names an owner and evidence location: a subscription read, secret-version record, fixture digest, database constraint, queue state, alert sample, reconciliation report, and recovery switch. Write facts and decisions in operational language, not internal task jargon in merchant-facing errors. The Liquid deep-practice guide can inform theme boundaries, while server-side webhook handling remains a separate acceptance surface. A failure case validates HMAC in a test shop and then reuses a live secret and global deduplication table. Multi-shop concurrency makes the error impossible to isolate. Recovery restores shop-scoped configuration and a clean test key space; the protected archive remains unchanged. If secret, topic, or shop mapping is unclear, isolate the entry point and investigate. A routine inspection starts with the oldest queue item and samples one success, retry, dead letter, and out-of-order event. Each should have raw bytes, verification result, business reference, and handler version. Group reports by topic and shop, not by a project label. For page context, the Shopify page-building trade-off guide is secondary; it cannot replace server-side webhook evidence.

The recovery boundary permits switching the receiver, signature adapter, deduplication rule, queue consumer, or one topic handler. It does not change product, order, inventory, customer, or notification data at the same time. Record the old rule, new rule, time, scope, and evidence for each switch. When the result is unclear, isolate the event instead of creating another external side effect.

After recovery, rerun the same fixtures and real samples and make signature, deduplication, ordering, retry, and dead-letter states explainable. Keep configuration and recovery snapshots before removing temporary switches. An acceptable result is not merely an empty queue: every event belongs to success, intentional skip, reconciliation, or review, with no unknown side effect.

Frequently asked questions

Why keep the raw request body?

HMAC is calculated over the bytes received. Parsing and reserializing can change whitespace, escaping, or field order, so the original body is needed to recompute a digest, explain a failure, and build a safe fixture. Protect it with access control and retention; logs should carry a digest and necessary references.

Is business idempotency still needed after webhook deduplication?

Yes. Webhook identity answers whether one delivery was seen, while a business idempotency key controls side effects such as an order, inventory, or notification write. A receipt can exist while its transaction is unfinished, so both layers need state and a result reference.

Should a subscription be removed after repeated failures?

Not as the first response. Classify the failure, protect raw bodies, preserve queue and dead-letter evidence, then choose a topic pause, delayed retry, or reconciliation query. A persistently failing subscription may be affected, so recovery includes re-registration and a data check for the outage window.

Can events be processed strictly by arrival order?

No. Store event time and object version, then use conditional updates or a current API reconciliation read. If versions cannot be compared, isolate the event. This prevents an old event from overwriting a newer fact or invoking an external action twice.

When may a real event be replayed?

Only when raw bytes, shop identity, signature, business key, and side-effect boundary are provable and current state has been checked. Start with a side-effect-free validation in isolation. Uncertain identity or result belongs in review instead of automatic replay.