On 15 September 2026 Cloudflare replaced its single "Block AI Bots" switch with separate Search, Training and Agent controls on every plan. Businesses using Cloudflare should open Security Settings, check what each control is set to, and avoid "Block" for search crawlers unless they truly want to vanish from Google and Bing.
Cloudflare sits in front of a large number of small business websites, often set up years ago by a developer or agency. That makes this a change worth checking even if you have never logged in to the Cloudflare dashboard yourself.
Key takeaways
- On 15 September 2026 Cloudflare introduced separate Search, Training and Agent controls for AI crawlers on all plans.
- A new Disallow AI Training option aims to keep sites in search while blocking AI training by mixed-use crawlers.
- Choosing Block can now affect Googlebot, Bingbot and Applebot, so a careless setting could hurt your search visibility.
- Cloudflare followed up in October with an OpenAI discovery pilot and a system that tracks whether verified bots respect robots.txt.
What happened
September: separate controls for search, training and agents
In a Cloudflare blog post on 15 September 2026, the company introduced controls for three types of bot activity: Search, Training and Agent. Each offers four options: Allow, Disallow AI Training, Block on pages with ads, and Block. The controls are available to all customers on all plans in each domain's Security Settings.
The key new option is Disallow AI Training, designed for "mixed-use" crawlers that serve both search and AI. It aims to keep a site in search results while opting out of training. Cloudflare also introduced an "Accountable" designation for bot operators that offer opt-outs, URL-level visibility and assurance that opting out of training will not affect search. It describes Apple, Google and Microsoft as Accountable.
Important details for existing sites:
| Situation | What Cloudflare says happens |
|---|---|
| Existing domains | Settings migrate automatically; old Training blocks become Disallow AI Training. "Nothing, in almost every case" is needed |
| Legacy features | Block AI Bots and Managed Robots.txt are being deprecated |
| Choosing Block or Block on pages with ads | Now applies to mixed-use crawlers such as Applebot, Bingbot and Googlebot, so it can affect search |
| New domains from 15 September | Onboarding offers presets; ad-supported sites get stricter defaults |
On Google specifically, Cloudflare notes that site owners can opt out of training by disallowing Google-Extended in robots.txt, and that Google states this does not affect search inclusion or ranking.
October: discovery and accountability
On 15 October 2026, Cloudflare and OpenAI shared early results from a research pilot started in July 2026. Using signals from about 1% of Cloudflare's network, the pilot led OpenAI to discover and index "hundreds of millions of new web pages". Cloudflare says the work uses metadata for search and discovery, "rather than sharing page content for model training", and that "a signal about a page doesn't grant permission to access it".
On 16 October 2026, Cloudflare described a system that continuously checks whether Verified bots honour robots.txt and slow down when a site returns HTTP 429 (Too Many Requests), feeding a trust rating it plans to publish on Radar.
Who this affects
- Any business whose domain runs through Cloudflare, including the free plan.
- Publishers and ad-funded sites, which now get stricter presets and a "Block on pages with ads" option.
- Businesses that want to appear in AI answers from ChatGPT, Copilot or Google AI features, where blocking can remove you from consideration.
- Agencies managing client Cloudflare accounts, who should audit settings across all domains.
The trade-off is a business decision, not a technical one. Blocking training protects your content; blocking search or agents can cost you visibility. Our guide on whether to block or allow AI crawlers walks through the options.
Practical checklist
- Log in to Cloudflare and open Security Settings for each domain.
- Check the Search control. For most small businesses it should be Allow. Block here can stop Googlebot and Bingbot.
- Decide on Training. If you do not want your content used to train AI models, choose Disallow AI Training rather than Block.
- Decide on Agents. AI agents may visit to book, compare or buy on a user's behalf. Blocking them may cost leads; allowing them adds load.
- Review your robots.txt. Since Managed Robots.txt is being deprecated, check what your live file says with our free robots.txt tester, and rebuild it with the robots.txt generator if needed.
- Test which AI crawlers can reach you with our free AI crawler checker.
- Watch for 429 errors in your logs or uptime monitoring if you tighten rate limits; legitimate search crawlers back off when they see them.
- Check Google's own AI controls. Search Console has a separate control for AI features; see our Search Console AI reports guide.
For broader Cloudflare housekeeping, see our Cloudflare free plan settings guide. To understand how AI search picks sources, read how ChatGPT search chooses sources.
FAQ
Will Cloudflare's new controls remove my site from Google?
Not by default. Cloudflare says existing settings migrate automatically and most domains need no action. However, choosing Block or Block on pages with ads now applies to mixed-use crawlers such as Googlebot and Bingbot, so those options can affect your search visibility. Check your Search setting.
What does Disallow AI Training actually do?
It is Cloudflare's option for keeping a site discoverable in search while opting out of AI training by crawlers that do both. It relies on bot operators honouring the signal, which is why Cloudflare introduced its Accountable designation for operators that commit to doing so.
Does blocking Google-Extended hurt my Google rankings?
According to Google, as cited by Cloudflare, disallowing Google-Extended in robots.txt does not affect search inclusion or ranking. It controls use of your content for some Google AI purposes. Google's AI features in Search are managed through Search Console's separate controls.
I use Cloudflare's free plan. Do these controls apply to me?
Yes. Cloudflare says the Search, Training and Agent controls are available to all customers on all plans, configured in each domain's Security Settings.
What is the Cloudflare and OpenAI pilot?
A research pilot started in July 2026 in which signals from about 1% of Cloudflare's network helped OpenAI discover new pages for ChatGPT search. Cloudflare says it uses metadata about pages rather than page content for training, and that signals do not grant permission to access a page.
Related reading
- Google's AI Content Guidance Update, October 2026: What UK Sites Should Change: Google updated its generative AI content guidance on 1 October 2026: fact-check everything, including titles, alt text and schema. What UK websites should do.
- Test robots.txt Rules Before They Cost You Traffic: A practical way to test robots.txt changes: how precedence and wildcards really match, per-crawler groups including AI bots, and which tools to trust.
Sources
Written by the MakeItFlow team
We build the website audit these guides are based on. Every fact is checked against the primary sources listed above and this guide was last reviewed on 25 October 2026 . How we research and check
Check your own site in 30 seconds
Run a free audit covering speed, SEO, DNS, email authentication and security headers — and get a prioritised fix list.
