AI Crawler Checker

An AI crawler checker tells you whether the bots behind ChatGPT search, Claude, Perplexity, Google AI Overviews and Microsoft Copilot are allowed to read your website. Many UK businesses block these crawlers by accident, through an old robots.txt rule, a security plugin or a CDN setting, and then wonder why AI assistants never mention them. Enter your domain and the check reads your live robots.txt and homepage, then shows which crawlers are blocked, which are allowed and what else could stop your pages being quoted.

How to use the AI crawler checker

1. 1 Sign in and open the toolLog in to your free account and open the AI Crawler Checker from the tools menu.
2. 2 Enter your domainType the domain only, such as example.co.uk. The checker fetches your homepage over HTTPS and the robots.txt file at the root of the site.
3. 3 Read the crawler groupsStart with the Googlebot and Bingbot result, then the AI search crawlers, then the training crawlers. Each one is marked as allowed or blocked.
4. 4 Check the page-level resultsLook at the snippet controls, the amount of text visible without JavaScript, the business identity markup and the llms.txt line.
5. 5 Fix and run it againCopy any suggested robots.txt lines or JSON-LD, update your site, clear your cache and rerun the check to confirm the change is live.

What an AI crawler checker actually tests

Every AI assistant that can browse the web relies on automated crawlers. Before a crawler fetches a page, a well-behaved one reads your robots.txt file and obeys the group of rules that names it, or the general User-agent: * group if no group names it. Our AI crawler checker applies those same rules to your homepage, so the result reflects what each crawler would be permitted to do today.

It then looks at the homepage itself. A crawler that is allowed in can still be stopped from quoting you by a nosnippet directive, or may see an almost empty page if your text is added by JavaScript. Finally it looks for structured data that says who the business is, and notes whether you publish an llms.txt file.

• Search engine crawlers: Googlebot and Bingbot, which feed Google AI Overviews, AI Mode and Microsoft Copilot as well as normal results.
• AI search crawlers: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude) and PerplexityBot (Perplexity).
• AI training crawlers and tokens: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot.

Why AI crawler access matters for AI search visibility

AI assistants increasingly answer questions such as "best accountant in Leeds" or "how do I fix a dripping tap" with a short summary and a handful of linked sources. If the search crawler for that assistant cannot fetch your pages, it has nothing of yours to cite, however good your content is. Being crawlable does not guarantee a mention, but being blocked all but guarantees you will not get one.

Google and Bing deserve special attention. Google AI Overviews and AI Mode are built on the normal Google index, and Copilot draws on Bing. There is no separate AI crawler to allow for these: if Googlebot or Bingbot is blocked, you lose both the classic results and the AI features on top of them. That is why the checker treats a blocked Googlebot or Bingbot as a failure and the rest as warnings or information.

How to read each result in the AI bot checker

Results use four statuses. A pass means nothing needs doing. A warning means something is likely costing you visibility. A fail means a problem serious enough to fix first. Information lines report a choice that is yours to make, so they never count against you.

The training crawler line is deliberately informational. Blocking GPTBot, ClaudeBot or Google-Extended stops those companies using your content to train future models; it does not remove you from ChatGPT search, Claude or Google Search. Google-Extended and Applebot-Extended are control tokens rather than separate crawlers: Googlebot and Applebot still visit, and the token simply records your preference about AI use.

• Googlebot and Bingbot can crawl the site (pass) or are blocked (fail).
• AI search crawlers allowed (pass) or some blocked (warning, with the exact robots.txt lines to add).
• Snippets switched off (warning) if a robots meta tag or X-Robots-Tag header contains nosnippet or max-snippet:0.
• Main text in the HTML: 150 words or more passes, 50 to 149 is a warning, fewer than 50 fails.
• Business identity marked up (pass) when Organization, LocalBusiness, a related subtype, Person or WebSite schema is found.
• llms.txt published or not published (information only).

How to unblock AI crawlers in robots.txt step by step

First, open yourdomain/robots.txt in a browser and find the groups that apply. A crawler follows only the most specific group that names it, so a group for GPTBot does not inherit anything from User-agent: *. If an AI search bot is blocked, either remove the rule that blocks it or add a dedicated group with Allow: / for it.

On WordPress, robots.txt is usually generated by your SEO plugin. In Yoast SEO go to Tools, then File editor; in Rank Math use General Settings, then Edit robots.txt. If a physical robots.txt file exists in the site root, it overrides the plugin, so edit that file instead. Some security and performance plugins also add bot-blocking rules, so check their settings too.

Shopify lets you change the file by adding a robots.txt.liquid template to your theme. Wix has a robots.txt editor under its SEO settings. Squarespace manages robots.txt for you, but has a setting to block known AI crawlers; switch it off if you want to be cited. Whatever the platform, also check your CDN or firewall: some services offer a one-click block of AI bots that works at network level and never shows in robots.txt.

Fixing snippet, JavaScript and schema problems

If snippets are switched off, search the page source for nosnippet and max-snippet. These are often set by a plugin setting or a theme option intended for one page and applied site-wide. Remove them unless you genuinely do not want Google to quote the page, because they also stop it appearing as a source in AI Overviews.

A low word count without JavaScript usually means a site built entirely in the browser, as some page builders and single-page apps are. Most AI crawlers do not run scripts, so ask your developer about server-side rendering or pre-rendering, and make sure key facts such as services, prices, opening hours and contact details are in the HTML itself.

For identity markup, add Organization schema with your name, URL and logo, or LocalBusiness if customers visit you. Yoast and Rank Math output this once you complete their site representation or local settings; Shopify themes and Wix add basic markup automatically, which you can extend with a JSON-LD block.

Best practice for AI crawlers on a UK business website

Decide your policy in two separate parts: whether you want to be cited by AI search tools, and whether you are happy for your content to be used for training. Most businesses that sell services or products want the first and are relaxed or undecided about the second. You can allow the search crawlers and block the training ones in the same file.

Keep robots.txt short and readable, review it whenever you change platform or install a security plugin, and run an AI crawler checker after any change. Treat llms.txt as an optional extra rather than a priority. The fundamentals that earn citations are the same ones that earn rankings: clear, accurate, useful pages that crawlers can read.

Examples you can copy

User-agent: *
Disallow:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.co.uk/sitemap.xml

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Plumbing Ltd",
  "url": "https://www.example.co.uk/",
  "logo": "https://www.example.co.uk/images/logo.png",
  "sameAs": [
    "https://www.linkedin.com/company/example-plumbing"
  ]
}
</script>

<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large">

Common mistakes to avoid

Frequently asked questions