Shopify Data Migration Engineering: Architecture, APIs, Idempotency, and Rollback

The conclusion: a migration system is not a one-off import script

Shopify data migration engineering should not end with a CSV or a loop that worked once. A production migration connects an immutable source snapshot, field contracts, object dependencies, identity maps, idempotent writes, throttled scheduling, row-level results, delta capture, business reconciliation, and rollback evidence. Every batch must answer what it read, which rule version transformed it, where it wrote, what failed, and whether replay is safe.

This guide owns the engineering implementation intent. Use the whole-store migration guide for Magento or Adobe Commerce program scope, SEO, and cutover planning. Use the customer migration guide for identity, consent, accounts, and privacy. Those boundaries prevent three useful pages from competing for the same query.

Freeze the engineering contract before writing calls

Version the migration specification: source systems and time zones, snapshot boundary, target shop, Shopify API version, app and scopes, object scope, exclusions, business keys, dependency graph, transform version, acceptance thresholds, freeze window, and rollback owners. Store it with code, schemas, queries, fixtures, and approvals.

Break “products” into product, option, variant, media, collection, metafield, inventory item, location quantity, and publication. Break “orders” into customer and merchandise references, currency, tax, discount, shipping, transactions, fulfillment, refunds, and historical-time semantics. Similar labels across platforms do not prove equivalent behavior.

Build a capability matrix for unsupported fields

For each object, record whether CSV, synchronous GraphQL, asynchronous mutation, migration app, or read-only archive is viable. Shopify's migration overview lists different paths for different objects and warns that product, customer, and historical-order sequence matters. Decide transform, downgrade, archive, or retirement before implementation.

Separate the control plane from the data plane

The control plane owns runs, batches, checkpoints, dependencies, retries, pauses, approvals, and reports. The data plane owns extract, normalize, transform, write, and result parsing. A single long script that also decides scope and deletes temporary files cannot reliably resume after interruption or compensate only the affected records.

Create an immutable run_id for every execution and a batch_id for every object slice. Store source_system + source_type + source_id + source_updated_at + transform_version + payload_hash per record. Store the target GID, mutation, API version, state, attempts, error class, and verification time for the result.

Reject illegal state transitions

A useful state machine is discovered → extracted → transformed → validated → submitted → applied → reconciled, with failures routed to retryable, quarantined, or rejected. An invalid transform cannot jump to submitted. A bulk job without a result asset cannot become applied. A batch without business reconciliation cannot become completed.

Create testable snapshots and data lineage

Production extraction needs a consistency boundary. Record maximum timestamp and ID, row count, predicate, source time zone, and file hash for every table or endpoint. Related tables need one logical snapshot or an explicit drift record. Keep raw, normalized, and Shopify-payload layers separate so a defect can be attributed to the source, transform, or target write.

Do not use one export timestamp as the only delta cursor. Child objects, consent, stock, media, or external systems can change independently. Choose CDC, timestamp plus stable ID, webhook stream, audit table, or temporary dual-write per domain. Advance a cursor atomically after batch settlement, never when a request is merely sent.

Expire sensitive snapshots automatically

Encrypt customer, order, and address files, restrict named access, audit downloads, and set deletion dates. Logs should carry source keys, target GIDs, field paths, codes, and hashes—not full contacts, tokens, addresses, or API headers.

Turn field decisions into executable contracts

A field contract defines source and target paths, type, length, enum, null behavior, default, transform, sensitivity, system of record, conflict priority, and acceptance query. Preserve currency with amounts, source zone with timestamps, an HTML allowlist with rich text, and a stable policy with handles and URLs.

Transforms should be pure: fixed input and rule version produce fixed output. Golden fixtures cover scripts and accents, emoji, combining characters, long text, nulls, duplicate SKUs, customers without email, multiple addresses, negative stock, tax-inclusive values, partial refunds, and missing media. Unknown enum values should fail or quarantine rather than silently becoming “other.”

Treat list fields as destructive writes

Set-style mutations can create, update, and remove list entries absent from the input. Verify current semantics in Shopify's productSet and customerSet references. A patch job that sends incomplete variants, metafields, or addresses can remove valid target data.

Orchestrate a dependency graph, not a filename order

Relationships dictate sequence. Definitions and locations usually precede products, options, variants, and media; customers precede orders; inventory waits for inventory-item and location maps; publications and extensions wait for their target GIDs. The exact graph follows the selected mutations and relationship policy.

Permit layer concurrency, not dependency races

Independent batches in one layer can run concurrently. Orders cannot write blindly while merchandise and customer maps are unsettled, and inventory cannot run before both GIDs exist. The scheduler derives runnable work from completed dependencies and current capacity instead of guessing with fixed sleeps.

Choose CSV, synchronous GraphQL, and Bulk Operations deliberately

CSV works for objects Shopify explicitly supports with simple, reviewable fields. Synchronous GraphQL fits smaller conditional writes that need immediate results. Bulk Operations fit large asynchronous queries or mutations. A program may mix them, but each object needs one authoritative write path.

Shopify's bulk import guide documents JSONL, stagedUploadsCreate, bulkOperationRunMutation, status tracking, and result retrieval. At this review, API 2026-01 and newer allow up to five bulk mutations per app per shop concurrently, input JSONL is capped at 100 MB, and a mutation operation must finish within 24 hours. Recheck those limits on implementation day; do not encode them as timeless constants.

A completed bulk job does not mean every row succeeded

Download the result JSONL, correlate __lineNumber or your key, and parse both transport-level problems and mutation userErrors. Reconcile successes, failures, and missing lines. Result URLs expire, so download them securely and hash them. A partialDataUrl can aid recovery, but partial output must be settled before rerunning the whole input.

Pin API versions, permissions, and authentication boundaries

Every request uses an explicit API version. Before an upgrade, run schema generation, compilation, contract tests, and the same fixtures in a development shop. Request only the scopes needed for current objects, and satisfy protected-customer-data requirements where applicable. Long-running backend jobs need an appropriate offline credential, not a staff member's session-bound online token.

Shopify's access-token documentation covers token types, lifetimes, refresh, and revocation. Public apps must also account for the 2027 Admin API change affecting non-expiring offline tokens. Store secrets in a secret manager and exclude plaintext credentials from logs, tickets, and snapshots.

Prove permissions in preflight

Read the shop, locations, required definitions, and a small object sample. Perform one removable write prototype and verify scopes. A 401, 403, wrong-shop response, unsupported version, or unavailable protected field blocks the run before bulk submission.

Migrate the product graph as a whole

Normalize product, option, and variant models first. Map old product ID, variant ID, SKU, and target GIDs separately because SKU can be blank or duplicated. Freeze approved handles; a rerun must not derive a new URL merely because a title changed.

productSet can synchronize a complete external product state and run synchronously or asynchronously, but its list semantics require a complete payload. Prototype option combinations, variant limits, metafield definitions, media processing, publications, and market visibility. Media request acceptance is not readiness: poll processing, validate order and alt text, and quarantine failed assets.

Separate product facts, price, stock, and publication

These domains often have different systems of record. One “product succeeded” count can hide an unpublished item, wrong market price, or zero inventory. Assign an owner, write path, and reconciliation to each domain.

Reuse dedicated identity and consent governance for customers

Use immutable source customer IDs as the primary map and normalized email or phone as conflict signals. customerSet can update by ID or upsert by email/phone, but contacts change, and address lists have set semantics. Persist the exact identifier, target GID, and every userError.

Marketing consent is not a generic Boolean. Preserve channel, state, source, time, and lawful basis, and use current dedicated consent mutations. Passwords generally do not move through customer CSV; invitation and activation messages need cutover, isolation, and support planning. The customer migration canonical owns those details.

Treat privacy deletion as a delta event

Deletion, merge, opt-out, and access requests after the full snapshot belong in the delta stream, not only create and update. Rollback and read-only archives must propagate applicable privacy actions too.

Preserve historical-order meaning without creating live side effects

Orders depend on customer and merchandise references, currency, tax, discount, shipping, and time. Shopify's orderCreate reference supports a historical processedAt for migration, subject to the current version and input rules. Define whether history serves operations, analytics, support, or an archive and how transactions, refunds, and fulfillments are represented.

Before importing, isolate new-order notifications, Flow, ERP, warehouse, email, loyalty, and finance webhooks. A historical paid or fulfilled object can still trigger a real downstream action. Prototype every state combination and document which automations pause or filter migration tags.

Reconcile an amount equation per currency

Validate line subtotal, discount, shipping, tax, refund, and total while preserving presentment and shop currency semantics. Use decimals or minor units, not binary floats, and agree on rounding before comparing values.

Decide the inventory source of truth before writing

Inventory depends on an inventory item and location. If the migration service represents the authoritative stock system, inventorySetQuantities can set absolute values and use compareQuantity for compare-and-set. Shopify's mutation reference warns that bypassing comparison can produce inaccurate quantities under concurrent writers. Evaluate adjustment semantics when the service is not authoritative.

Never hide location drift with a matching total

The same aggregate at the wrong warehouse changes promise and fulfillment behavior. Reconcile source item plus source location to inventory-item GID plus location GID.

Design idempotency, deduplication, and retry together

A durable identity combines source system, object type, immutable source ID, and business version. Consult the identity map before writing and persist the target GID atomically after confirmation. If a request times out after possible execution, query by map or stable business identity before retrying. Never assume timeout means no write.

Retry transient network, recoverable 5xx, and explicit throttle failures with bounded exponential backoff and jitter. Validation, permission, conflict, and missing-dependency failures go to quarantine. Preserve every attempt and original error.

Compare normalized payload hashes semantically

Canonicalize field order, null behavior, and time before hashing. The same hash can skip; a changed hash still needs system-of-record and target-version checks. A newer source timestamp alone must not overwrite a valid post-launch Shopify edit.

Schedule from returned cost and bulk state

Synchronous GraphQL uses query cost. Read requested and actual cost plus throttle state and adjust concurrency dynamically. Shopify's API limits guide recommends bulk operations for large reads. Fixed requests-per-second logic cannot reflect mutation cost or shop capacity.

Bulk queries avoid client pagination for large exports. The bulk query guide covers status, object count, cancellation, and result assets. Store every operation ID. An observation timeout means query the same operation again, not start a duplicate job.

Stop instead of retrying forever

Pause on repeated authentication failure, excessive user errors of one class, missing dependencies, P0 corruption, widening differences, platform-duration risk, or unavailable results. Keep the checkpoint and resume only from a proven boundary.

Reconcile five layers: count, field, relation, value, and behavior

Counts prove the equation closes. Field checks find nulls, truncation, enums, and encoding. Relation checks find product-variant, customer-order, and inventory-location orphans. Value checks cover money, currency, refunds, and stock. Behavior checks ask real operators to search, sign in, order, refund, find history, and fulfill.

Produce a machine-readable difference pack

Every difference includes source key, target GID, field path, source and target hashes, severity, owner, and disposition. A colored summary alone is insufficient; engineers need to drill into records and replay a bounded set.

Use high-water marks and catch-up windows for cutover

Capture a high-water mark before the full load and keep collecting changes. Run deltas toward newer marks until volume and duration fit the freeze. During final freeze, restrict source writes, settle the last delta, execute fast reconciliation, then enable Shopify writers and traffic. External systems that cannot freeze need a queue, dual-write, or conflict precedence.

The cutover checklist also covers DNS, domain, URL redirects, payment, tax, shipping, stock writer, order writer, webhooks, ERP, marketing, account email, and monitoring. “The API finished” is not a business launch gate.

Order deltas with business clocks

Record source event time, source sequence, ingested time, and applied time. Detect clock drift, late events, and identical timestamps. Use time plus stable ID or a native sequence, overlap the cursor window, and deduplicate.

Rollback is a compensation protocol, not a delete command

Define triggers such as wrong prices, overwritten identity, stock drift, live order side effects, missing URLs, unexplained reconciliation, or a critical downstream outage. Specify the order for pausing writers, restoring traffic, reverse-syncing new orders and customers, compensating bad batches, restoring automation, reconciling, and communicating.

Batch markers identify scope but do not authorize blanket deletion; Shopify may contain valid post-cutover orders, edits, or stock events. Define a reversible update, compensating mutation, re-import, or manual action for each mutation family. Exercise one real failure in a rehearsal shop and measure recovery time.

A backup is not an operational rollback

A source snapshot reconstructs old state but cannot absorb target-side transactions or recall email and warehouse actions. Recovery evidence must cover every writer and external side effect.

Make observability useful without leaking data

Track throughput, latency, GraphQL cost, throttle, bulk state, object success, userError class, retry, quarantine, difference, delta lag, and backlog. Logs carry run, batch, source ID, target GID, operation ID, trace ID, and transform version. Alerts link to runbooks and distinguish pausing an object from stopping the program.

Mask structured fields and expire controlled error samples. APM, log vendors, tickets, and support tools are part of the data flow and belong in access and deletion inventories.

Publish a dashboard without personal data

Show progress, error classes, differences, and SLA, not customer plaintext. Business owners get a go/no-go view while engineers use an audited tool for record-level inspection.

A 30-day engineering route

Week one freezes objects, version, scopes, fields, dependencies, fixtures, and a small prototype. Week two builds immutable extraction, transforms, identity maps, result records, and synchronous writes. Week three adds bulk jobs, throttling, delta, reconciliation, monitoring, and quarantine. Week four runs two full rehearsals plus delta, load, notification isolation, fault, and rollback tests and obtains technical and business approval.

Timing varies with volume, source quality, markets, subscriptions, B2B, and integrations. WESWOO's migration services and Shopify store setup guides provide implementation context. The invariants are stronger than a calendar: evidence for every input, traceability for every write, an owner for every difference, and rehearsed cutover and rollback.

Frequently asked questions

Should a Shopify migration use CSV or GraphQL first?

Choose per object. CSV fits explicitly supported and simple reviewable imports, synchronous GraphQL fits conditional smaller writes with immediate results, and Bulk Operations fit large asynchronous work. Familiarity with one tool is not a reason to force all objects through it.

Does completed mean a Bulk Operation migrated every record?

No. Download and parse the result JSONL, correlate input lines, inspect mutation userErrors, missing lines, and partial output, then reconcile objects and business behavior. Job state alone is not quality evidence.

How do we prevent duplicates when a run is repeated?

Persist stable source IDs and source-to-GID mappings, detect normalized conflicts, hash canonical payloads, and update conditionally. After a timeout, query whether the write already happened before issuing another create.

Can the worker sleep for a fixed time after throttling?

Fixed sleeps ignore GraphQL cost, shop capacity, and other jobs. Read throttle state, tune concurrency, use bounded exponential backoff, evaluate bulk operations for scale, and pause when error thresholds are crossed.

When must a migration batch stop?

Stop on wrong shop or permissions, identity or consent overwrite, money or stock corruption, broken critical relations, live notification side effects, unexplained differences, missing result evidence, or an unexecutable rollback. Preserve the checkpoint, repair, and resume from the last proven boundary.