Search workflow

Catch a crawler rule that changed after launch.

A site redesign can quietly change which crawlers can retrieve public pages. Suede’s AI Visibility Watch reads robots.txt on a weekly schedule and returns the affected crawler groups with a short alert.

Open AI Visibility Watch

Repeat the same question and record the observation.

Illustrative watch result

A shop moves to a new host

Input: The public shop URL and the merchant’s approved access policy.
Expected output: A robots status, crawler groups and a short alert for the reviewed host.
Inspect: Compare the flagged path with the public catalog path.
Decision: Correct an accidental catalog restriction or record the intended exclusion.

Choose the host and write down its intended policy

Enter the public site URL in the template. Confirm whether the relevant resource lives on the main domain, a shop or another subdomain. Keep the business’s intended search and training permissions nearby so the reviewer knows which restrictions were deliberate.

Read the fields in the alert

The workflow returns the site, the robots response status, separately listed answer and training bots, an alert and a summary. Review the actual user-agent group and path rule on the live host before assigning a repair. A change to one path can leave the rest of the site accessible.

Decide who owns the change

An unexpected block on a public service should go to the owner of that site’s publishing or delivery configuration. If the rule matches the approved policy, record the review date. After a repair, retrieve the published rule and a representative public URL again.

Catch a crawler rule that changed after launch.

A shop moves to a new host

Choose the host and write down its intended policy

Read the fields in the alert

Decide who owns the change

Explore related work