Shopify robots.txt should reduce clearly low-value crawling paths while keeping products, collections, articles, and important language pages discoverable. It cannot reliably prevent indexing after a URL is referenced, and it does not replace noindex, canonicals, authentication, or deletion. Preserve the current file, goal, and rollback before changing it.
Classify paths by intent
List home, product, collection, article, search, filter parameters, cart, checkout, account, internal API, and test paths. For each, record crawl need, index need, genuine search intent, and sitemap status. Do not Disallow a path merely because it looks complex, and do not block CSS or JavaScript needed for rendering.
Validate with Search Console and logs
Before release, inspect robots text, sitemap declaration, status, canonical, and rendered HTML. After release, compare server logs and Search Console crawl data to ensure priority market and language pages remain visited while low-value parameters do not consume crawl. Ship rules in stages and record date, owner, affected URLs, and rollback condition.
FAQ
Does robots.txt prevent indexing?
Not reliably. Use noindex, authentication, removal, or canonical controls for indexing decisions.
Can every parameter URL be blocked?
Not automatically. Determine whether parameters carry sorting, filtering, or market intent before rules and canonicals.
Why should CSS and JavaScript stay crawlable?
Search systems need resources to understand rendered content; blocking them can impair interpretation.
What should be monitored after a change?
Logs, Search Console crawl, priority-page indexing, rendering, 404s, and sitemap coverage.