Block or Allow AI Crawlers? GPTBot, ClaudeBot and More

Deciding whether to block AI crawlers is now a routine website decision, not a niche one. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and a handful of others all read your robots.txt, but they do very different jobs. Some collect training data, some fetch pages so an AI assistant can cite you, and blocking the wrong one can quietly remove you from AI search answers. This guide explains what each crawler does, what blocking it actually changes, and a sensible default policy for UK businesses.

Training crawlers vs AI search crawlers

AI companies now split their bots by purpose, and that split is the key to a good policy.

• Training crawlers collect content that may be used to train future AI models. Blocking them is an opt-out from training. It does not remove you from search.
• Search crawlers fetch and index pages so an AI assistant can show and link to them in answers. Blocking them makes you invisible in that assistant's search results.
• User-triggered fetchers visit a page because a person asked the assistant to look at it. These are treated more like a browser than a crawler, and some do not follow robots.txt.

The practical upshot: you can block training and still be cited in AI search. OpenAI, Anthropic and Perplexity all document their bots as independent settings.

Every major AI crawler at a glance

OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User

OpenAI's documentation is explicit: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models," while "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." The two settings are independent, and if you allow both, OpenAI may use a single crawl for both purposes. OpenAI says it can take around 24 hours for its systems to pick up a robots.txt change.

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

Anthropic runs three bots. ClaudeBot collects content that could contribute to model training. Claude-SearchBot indexes content to improve Claude's search results, and blocking it "may reduce your site's visibility" there. Claude-User fetches pages when a Claude user asks a question that needs them. Anthropic says its bots respect robots.txt and recommends robots.txt over IP blocking, which "may not work correctly or persistently". It also supports the non-standard Crawl-delay directive if you want to slow ClaudeBot down rather than block it.

Perplexity: PerplexityBot and Perplexity-User

PerplexityBot surfaces and links websites in Perplexity's results and, according to Perplexity, "is not used to crawl content for AI foundation models." Perplexity-User fetches pages when a user asks a question and generally ignores robots.txt because the request was user-initiated. Changes can take up to 24 hours to apply.

Google-Extended

Google-Extended is a robots.txt token, not a separate crawler. Google crawls with its normal user agents and uses the token to decide whether that content may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Note what it does not cover: AI Overviews and AI Mode are part of Google Search and are fed by Googlebot. Google now offers a separate site-level control in Search Console for those features, which we covered in our guide to the new Search Console reports.

Applebot-Extended

Like Google-Extended, Applebot-Extended does not crawl anything. Apple says it only governs whether content collected by Applebot can be used to train its foundation models. Pages that block Applebot-Extended can still appear in Apple's search features such as Siri and Spotlight suggestions, and the rule is not used in ranking.

CCBot

CCBot is run by Common Crawl, a non-profit that publishes an open archive of web crawl data that anyone, including AI developers, can download. Blocking CCBot keeps future crawls of your site out of that archive. Common Crawl warns that some bots falsely claim to be CCBot and publishes its IP ranges so you can check.

robots.txt examples for AI crawlers

robots.txt is a request, not a lock. Reputable AI companies say they honour it, but nothing technically stops a badly behaved scraper. Each group below starts with a User-agent line, and crawlers follow the most specific group that names them.

Block AI training, allow AI search (recommended default)

# AI training crawlers: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

# AI search crawlers: allow so assistants can cite you
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Everyone else
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap.xml

Allow everything

If you want maximum AI visibility, you do not need AI-specific rules at all. A crawler with no matching group falls back to User-agent: *.

Block all AI crawlers

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Never add Googlebot or Bingbot to an AI block list. They power Google Search, AI Overviews, Bing and Microsoft Copilot, so blocking them removes you from classic search too. If you are unsure how groups and wildcards interact, our robots.txt guide covers the syntax, and the free robots.txt generator builds a file in your browser with an optional AI training opt-out.

Platform notes

• WordPress: many SEO plugins (Yoast, Rank Math, All in One SEO) let you edit robots.txt from the dashboard. If a physical robots.txt file exists in the web root, it overrides the virtual one WordPress generates.
• Shopify: edit the robots.txt.liquid template under Online Store > Themes > Edit code.
• Wix and Squarespace: both generate robots.txt for you; check their current settings for AI crawler toggles before trying to edit the file directly.
• cPanel/Apache and Nginx: upload or edit robots.txt in the document root of each hostname, including subdomains.

Cloudflare AI bot settings

If your site runs through Cloudflare, you have options beyond robots.txt, and they have changed recently.

1. In the Cloudflare dashboard, open Security Settings and choose Configure AI bot policies.
2. Set a policy for each category: Training, Search and Agent. Each can be Block (all pages), Block on pages with ads, or Allow.
3. Optionally turn on Bot Preference Sync, which adds a managed robots.txt with Disallow rules for known training crawlers. If you already have a robots.txt, Cloudflare prepends its rules above yours and serves them as one file.

Three points worth knowing:

• The legacy "Block AI bots" toggle is being deprecated on 15 September 2026. Cloudflare says that from that date, new domains block Training and Agent bots on pages that display ads by default, while Search stays allowed.
• Robots.txt is a preference; Cloudflare's blocks are enforcement. Cloudflare recommends using robots.txt to state your preference and AI Crawl Control to enforce it, and AI Crawl Control can show crawlers that requested paths your robots.txt disallows.
• Check the Search category is not blocked unless you mean it. Blocking Search crawlers removes you from AI assistants' cited results.

A recommended AI crawler policy for small businesses

For most small businesses, service firms, shops and local trades, this is a sound starting point:

1. Allow Googlebot and Bingbot. Non-negotiable if you want search traffic.
2. Allow AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Being cited in ChatGPT, Claude and Perplexity answers is a growing source of referrals and brand exposure.
3. Decide on training crawlers deliberately. Blocking GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot does not cost you search visibility. If your content is your product, such as original research, journalism or paid courses, blocking makes sense. If you mainly want to be known and recommended, allowing training may help models learn about your brand.
4. Write the decision down with the date, and review it every six months. These bots and their documentation change often.
5. Test after every change. A single misplaced Disallow: / under User-agent: * blocks everything.

Publishers with paywalled or licensed content may want to go further with Cloudflare enforcement. For everyone else, robots.txt plus the default above is enough. If you are also thinking about AI-friendly content files, see our explainer on llms.txt, which is a separate topic from crawler access. If you publish one, the llms.txt checker validates the format and tests its links.

How to check what your site currently allows

Our free AI crawler checker reads your live robots.txt and reports, separately, whether Googlebot and Bingbot can crawl the site, whether the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are allowed, and which training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot) are blocked. It also checks snippet controls such as nosnippet, how much text is readable without JavaScript, Organization or LocalBusiness schema, and whether you publish an llms.txt file. It needs a free account. To check robots.txt alongside titles, sitemaps, speed and security headers, run a free website audit.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, so you can block GPTBot and still appear in ChatGPT search answers as long as OAI-SearchBot is allowed.

Does blocking Google-Extended affect my Google rankings or AI Overviews?

Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews are part of Search and rely on Googlebot, so Google-Extended does not control them; Search Console has a separate setting for that.

Do AI crawlers actually obey robots.txt?

OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl all document robots.txt controls for their crawlers. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt, and unknown scrapers can ignore it entirely, which is why Cloudflare offers enforced blocking.

How long does a robots.txt change take to work for AI bots?

OpenAI says around 24 hours, and Perplexity says up to 24 hours. Other crawlers pick up changes the next time they fetch your robots.txt, so allow a day or two before testing.

Should a small business block AI crawlers?

Most should allow AI search crawlers so assistants can cite them, and make a deliberate choice about training crawlers. Blocking training bots costs no search visibility, so it is a reasonable choice if you are uneasy about your content being used for model training.

Related reading

• Cloudflare's New AI Crawler Controls: What Websites Should Check: Cloudflare replaced Block AI Bots with Search, Training and Agent controls on 15 September 2026. What sites should check to stay visible in search.

Sources

• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity: Perplexity crawlers
• Google Search Central: Google's common crawlers (Google-Extended)
• Apple: About Applebot
• Common Crawl: CCBot
• Cloudflare Docs: Block AI bots
• Cloudflare Docs: Managed robots.txt