Technical SEO audit of a Shopify store
(Only partial insights shared for client privacy)
01What we found
Pages crawled
17,286
11,976 product, 5,233 category, 74 non-200 responses in total.
Pages blocking AI crawls
0
Across 38 named AI agents and 17,286 URLs tested. Access is not the problem.
At risk of truncation
62%
173 of 277 pages fetched, then cut off before the content is reached.
AI readiness across 277 sampled pages
Scored on delivery, payload and structure · median 68.1
02What to do first
Ordered by what moves the most, and by what has to happen before it.
- Cut page weight. The median page is 1.9 MB and 2.9% of it is text. That ratio is why 173 of 277 sampled pages are at risk of being cut off before an AI crawler reaches the product information: it works to a token budget and stops. A search crawler will read the whole page, but it spends crawl budget doing it and the speed counts against the page. Defer scripts, trim the markup the theme emits, lazy-load media. It is also the only item here that makes the page faster for shoppers. Applies to: AI crawlers first, search crawlers second. Effort: an engineering sprint. Impact: highest here.
- Add cache headers. No page returns Cache-Control, ETag or Last-Modified, so every return visit by either crawler downloads two megabytes again instead of ending in a 304. Adding them is a platform-level change measured in hours. Applies to: AI and search crawlers equally. Effort: hours. Impact: high, and it repeats on every crawl.
- Declare the variant relationship on product pages. 11,976 product pages publish eight offers each with nothing tying them together. A search crawler sees eight products competing for the same query; an AI assistant answering a size or colour question has no way to know they are one item. ProductGroup with hasVariant fixes both in a single template. Applies to: AI and search crawlers. Effort: one template. Impact: the most valuable schema change here.
- Support the 10,153 star ratings with review markup. The rating is marked up. The reviews behind it are not. Search crawlers will not show a review snippet on an aggregate with nothing underneath it, and an AI assistant asked whether a product is well reviewed has a number and no evidence. Applies to: search crawlers first, AI assistants second. Effort: depends on the review app. Impact: medium.
- Turn off the collection generator before pruning anything. The 1,425 empty collections exist because the platform builds one for every tag combination. Prune first and they return next month. Turn it off, keep the few with real search demand, redirect the near-duplicates, noindex the rest. Both crawlers currently spend requests on pages that answer with nothing. Applies to: AI and search crawlers. Effort: one setting, then a week of triage. Impact: crawl efficiency.
- Add ItemList to the collection template, Organization and WebSite to the homepage. ItemList tells both crawlers that a category page is a ranked list rather than a wall of links. Organization and WebSite give the brand an entity for a search crawler to attach to and for an AI assistant to resolve the company against. Two template edits covering 5,233 category pages and the homepage. Applies to: AI and search crawlers. Effort: an afternoon. Impact: modest.
03The data behind it
What the crawler actually receives
277 sampled pages, measured under crawler conditions
| Measure | Median | 90th pct | Worst | Reading |
|---|---|---|---|---|
| HTML delivered | 1,092 KB | 1,911 KB | 2,360 KB | Sitewide the average page is 1,922 KB and the largest 4,199 KB. |
| Text as a share of HTML | 2.93% | — | 1.91% | On the homepage, 14 KB of text arrived inside 1,356 KB of markup. |
| Where content starts | 36.6% | 39.2% | — | A third of the document is consumed before the first useful word. |
| Time to first byte | 1,211 ms | 1,954 ms | 2,976 ms | Marginal for a crawler working 17,000 URLs, and over two seconds on 20 pages. |
| Cache headers present | 0 of 277 | — | — | No Cache-Control, no ETag, no Last-Modified anywhere. |
173 of 277 pages are flagged at risk of truncation, and 20 are heavy enough that a crawler is expected to abandon them before finishing. Every one of those pages returned a clean 200.
Structured data: expected against found
Schema validation pass over 17,286 pages
| Type | Pages expecting it | Found | Valid | Gap |
|---|---|---|---|---|
| Product | 11,976 | 11,976 | 11,975 | Complete. One warning across the set. |
| BreadcrumbList | 17,285 | 11,976 | 11,976 | Missing on 5,309 pages, all of them category pages. |
| ItemList | 5,233 | 0 | 0 | No category page declares that it is a list of products. |
| ProductGroup | 11,976 | 0 | 0 | Eight offers per product, no relationship declared between them. |
| Review | 10,153 | 0 | 0 | The ratings are marked up. The reviews behind them are not. |
| Organization | 1 | 0 | 0 | A partial node appears on 11,979 pages; none of it validates. |
| WebSite | 1 | 0 | 0 | Nothing on the site has a company entity to attach to. |
| FAQPage | 1 | 0 | 0 | The FAQ page carries no FAQ markup. |
The product markup is close to perfect and that is the number that gets quoted. It measures whether the markup that exists is well formed, not whether the right types are present.
Validation status by page type
The same pass, counted by page rather than by type
| Page type | Pages | Valid | Errors | Share valid |
|---|---|---|---|---|
| Product | 11,976 | 11,975 | 0 | 99.99% |
| Category | 5,233 | 0 | 5,233 | 0% |
| Other templates | 75 | 0 | 75 | 0% |
| Homepage | 1 | 0 | 1 | 0% |
| FAQ | 1 | 0 | 1 | 0% |
| All pages | 17,286 | 11,975 | 5,310 | 69.3% |
Read by page type instead of by markup quality, the picture inverts: every category page on the site fails validation, and category pages are where a shopper starts.
Why 1,425 pages were classified as empty
5,213 candidates screened by page size, structure and rendered content
| Classification | Pages | What the page returns |
|---|---|---|
| Empty collection | 845 | The rendered page states explicitly that it holds zero products. |
| Populated shell | 575 | A grid renders, but the body carries only 11 product links and no collection content. |
| Thin body | 5 | 166 useful bytes after boilerplate is removed, and 26 content words against a threshold of 50. |
| Classified as empty | 1,425 | All returning a clean 200 at between 1.2 and 2.2 MB. |
None of these is an error page. They return 200, they are indexable, and they weigh two megabytes each. Nothing in a standard crawl report flags them.
Every named AI agent, checked against robots.txt
38 agents tested · 24 shown · 0 fully blocked
| Agent | Company | Purpose | robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Training crawler | Allowed |
| ChatGPT-User | OpenAI | ChatGPT browsing | Allowed |
| OAI-SearchBot | OpenAI | ChatGPT search | Allowed |
| ClaudeBot | Anthropic | Training crawler | Allowed |
| Claude-User | Anthropic | Claude browsing | Platform defaults |
| Claude-SearchBot | Anthropic | Claude search | Platform defaults |
| Googlebot | Search crawler | Platform defaults | |
| Google-Extended | Gemini training | Allowed | |
| Google-CloudVertexBot | Vertex AI | Platform defaults | |
| bingbot | Microsoft | Bing and Copilot | Allowed |
| PerplexityBot | Perplexity | AI search | Allowed |
| Perplexity-User | Perplexity | AI assistant | Allowed |
| Applebot | Apple | Search and Siri | Allowed |
| Applebot-Extended | Apple | Apple Intelligence | Allowed |
| Meta-ExternalAgent | Meta | Meta AI training | Allowed |
| Meta-ExternalFetcher | Meta | Meta AI assistant | Platform defaults |
| Amazonbot | Amazon | Alexa answers | Allowed |
| Bytespider | ByteDance | Doubao training | Allowed |
| DuckAssistBot | DuckDuckGo | AI assistant | Allowed |
| CCBot | Common Crawl | Open training dataset | Allowed |
| MistralAI-User | Mistral | AI assistant | Platform defaults |
| cohere-ai | Cohere | Training crawler | Allowed |
| YouBot | You.com | AI search | Allowed |
| ai2bot | Allen Institute | OLMo training | Allowed |
Of 38 agents checked, none is fully blocked. The 17 marked platform defaults inherit the store platform's wildcard rules, which cover cart, checkout, account and search parameters and no product or category content. Across 17,286 URLs tested, zero were blocked to any agent.