Cloudflare OpenAI AI Discovery Pilot: October 2026 Guide

On October 15, 2026 Cloudflare and OpenAI shared results from a pilot that uses Cloudflare's network signals to help ChatGPT search find new and updated pages while making fewer wasted requests. A day later Cloudflare began scoring Verified bots on whether they respect site preferences. Sites wanting ChatGPT visibility should allow OpenAI's search crawler and remove crawl traps.

Both announcements point the same way: AI search is starting to behave more like traditional search, with discovery signals, crawl efficiency and rules about bot behavior. That is good news for small sites that are hard for AI systems to find.

Key takeaways

On October 15, 2026 Cloudflare and OpenAI described a pilot, running since July 2026, that uses Cloudflare network signals to help OpenAI's search crawler find new and changed pages.
Using about 1% of Cloudflare's network, the pilot helped OpenAI discover and index hundreds of millions of new pages, and it filters out wasted requests such as duplicate URLs.
On October 16, 2026 Cloudflare added continuous trust checks for Verified bots, based on whether they respect robots.txt and slow down after HTTP 429 responses.
Sites that want ChatGPT search visibility should allow OAI-SearchBot, avoid redirect and duplicate traps and keep robots.txt deliberate.

What happened

The discovery pilot

In a Cloudflare blog post dated October 15, 2026, written by Jack Galilee of Cloudflare and Henry Aspegren and James Balfour of OpenAI, the companies describe a research pilot launched in July 2026. Signals from Cloudflare's network help OpenAI's search crawler find useful pages. OpenAI then independently evaluates and ranks those pages.

Key points from the post:

Scale. The pilot used a cohort of about 1% of Cloudflare's network and helped OpenAI discover and index "hundreds of millions" of new web pages. The authors say "the pilot has brought previously unknown pages into OpenAI's search index".
Less waste. Cloudflare identified patterns that waste crawls, such as bots requesting files that do not exist, sites that redirect almost any address to the homepage, and different URLs serving the same content.
Change signals. These tell a crawler when a page is worth revisiting, so it does not keep downloading pages that have not changed.
Harmful pages. Signals are being tested to flag a single harmful page, such as a fake login form on an otherwise legitimate site, without penalizing the whole domain.

The companies stress that the research "uses metadata about the web for search and discovery, rather than sharing page content for model training", and that "a signal about a page doesn't grant permission to access it." The post notes that new discovery may briefly increase crawling while pages are evaluated. The post does not describe a sign-up process for site owners.

Continuous accountability for Verified bots

On October 16, 2026 Cloudflare published Continuous accountability for Verified bots. Being on Cloudflare's Verified bot list proves a bot's identity, but not that it behaves well. Cloudflare will now track behavior over time and give each bot an "explainable trust band". The first two signals are:

Signal	What Cloudflare checks
robots.txt	Whether a bot's traffic stays consistent with the site's published rules over time, looking for repeated patterns rather than one-off slips
HTTP 429 (Too Many Requests)	Whether a bot slows down or pauses after being told to, or carries on at the same pace

Cloudflare is using the system internally for now and plans to let all customers choose the trust threshold for how Verified bot traffic is handled. It also plans to publish scores on Cloudflare Radar. No dates are given.

Who this affects

Publishers, bloggers and content-led businesses that want to be cited in ChatGPT search answers.
Small and new sites with few backlinks, which are the pages a crawler is least likely to find on its own.
Sites on Cloudflare, which can use its bot controls and, later, the trust threshold.
Everyone with a robots.txt file, because bot behavior is now being measured against it.

How OpenAI's crawlers differ

OpenAI's crawler documentation lists separate user agents with separate robots.txt controls:

OAI-SearchBot is for search. Sites that block it "will not be shown in ChatGPT search answers", though they may still appear as navigational links.
GPTBot crawls content that may be used to train OpenAI's models. Blocking it signals your content should not be used for training.
ChatGPT-User visits pages when a user asks ChatGPT something. Because users trigger these visits, OpenAI says robots.txt rules may not apply.

You can allow OAI-SearchBot and block GPTBot at the same time. Our guide to blocking or allowing AI crawlers walks through the options.

Practical checklist

Decide your AI policy. Search visibility and model training are separate choices. Write down what you want before editing anything.
Check which AI bots you allow. Our free AI crawler checker shows how your robots.txt treats OAI-SearchBot, GPTBot and others. Test specific rules with our robots.txt tester.
Check Cloudflare settings if you use it. Make sure firewall or bot rules are not blocking crawlers you want. Our post on Cloudflare's AI crawler controls explains the dashboard.
Remove crawl traps. Cloudflare named catch-all redirects to the homepage and duplicate URLs as waste. Run key URLs through our redirect checker and make sure missing pages return a real 404. See our guide to soft 404 errors.
Fix duplicate URLs. Use canonical tags and consistent internal links, as explained in our guide to duplicate content.
Keep your sitemap accurate. Accurate last-modified dates are a clear change signal for any crawler. Check yours with our sitemap checker.
Use 429 responses, not blanket blocks, for heavy crawlers. If a bot is overloading your server, a rate limit with an HTTP 429 response is the signal Cloudflare now tracks.

For how ChatGPT picks which pages to cite once it finds them, read how ChatGPT search chooses sources. Make It Flow can also help you automate monthly checks of robots.txt, redirects and sitemaps.

FAQ

What is the Cloudflare and OpenAI discovery pilot?

It is a research pilot, launched in July 2026 and described on October 15, 2026, in which signals from Cloudflare's network help OpenAI's search crawler find new and updated pages and avoid wasted requests. OpenAI still decides independently which pages to rank in ChatGPT search.

Does Cloudflare share my website content with OpenAI?

According to the companies, the research uses metadata about the web for search and discovery rather than sharing page content for model training. They also say a signal about a page does not grant permission to access it, so your robots.txt and Cloudflare settings still apply.

Do I need to sign up to be part of it?

The October 15 post does not describe a sign-up process for site owners. It points to Cloudflare's AI Crawl Control for managing how AI systems access your site. To appear in ChatGPT search, the key step is allowing OAI-SearchBot in robots.txt and firewall rules.

What are Cloudflare Verified bots?

They are automated crawlers whose identity Cloudflare has confirmed, such as major search engine bots. From October 16, 2026 Cloudflare is also tracking whether they respect robots.txt and slow down after HTTP 429 responses, and assigning each one a trust band that can move up or down.

Should I block GPTBot but allow OAI-SearchBot?

That depends on your goals. OpenAI says the two are controlled separately: OAI-SearchBot affects whether you appear in ChatGPT search answers, while GPTBot relates to model training. Many businesses allow search and decide on training separately. Neither choice affects your Google rankings.

Cloudflare and OpenAI AI Discovery Pilot: What It Means for Your Website