Research

What can a crawler study tell you?

Suede AI’s September 20, 2026 AI Crawler Access Index examined 1,000 domains in a fixed Tranco list. For a marketing team, the useful question is whether the evidence describes a site’s stated policy, an actual visit or a citation in a buyer’s answer.

Read the AI Crawler Access Index

September 20, 2026 · Suede AI Crawler Access Index.

September 2026 study

Two findings with different denominators

138 of 497: Evaluated policies disallowing the root for one or more selected tokens.
78 of 1,000: Domains with an llms.txt candidate meeting the study’s format and control checks.
Next question: Which rule applies to the exact public page and provider role you care about?

Explore research and evidence

Filter by topic4 guides

Read the denominator first

The team could evaluate the root-path policy for 497 domains. Of those, 138 disallowed the root for at least one of five selected AI-related tokens. The other 503 domains were unclassified for this measure. Treating those as open or closed would change the finding.

Check the decision your team actually controls

Pick a public product or service page. Record the exact hostname, path and crawler role. A rule on the homepage can differ from a rule on a subdirectory. Search access and model-development permissions also serve different business decisions.

Keep the observations reproducible

The study includes the sample, method and downloadable observations. For your own review, keep the URL, timestamp and response beside the rule you interpreted. Use a separate dated prompt run to establish whether an assistant cited the page.