AI Crawler Blocking Study 2026: We Checked the Top 10,000 Websites

How do the world's most popular websites treat AI crawlers? On 29 October 2026 we checked the 10,000 most popular domains in the Tranco research ranking. For each one we looked at robots.txt rules for 16 AI user agents, llms.txt, what the server does when a request identifies itself as an AI search bot, HTTPS and security headers, and SPF and DMARC email records.

The short version: 22.4% of sites with a robots.txt block at least one AI crawler. Almost all of them target training crawlers, and only 9.3% block an AI search crawler. A growing number of sites also refuse AI bots at the server rather than in robots.txt, so robots.txt alone undercounts AI blocking. Meanwhile 14.0% of live sites already publish an llms.txt file, and only half of all domains enforce DMARC.

All figures are aggregates. We do not publish a list of individual sites. A few very large, well-known sites are named only as examples of a policy.

Key findings at a glance

• 22.4% (990 of 4,425) of sites with a valid robots.txt block at least one of 16 AI user agents from the whole site while still allowing Googlebot. Among the top 1,000 the share is 27.6% (134 of 485). Among ranks 1,001 to 10,000 it is 21.7% (856 of 3,940).
• Training crawlers are blocked about twice as often as AI search crawlers. 22.0% block at least one training crawler, but only 9.3% block an AI search crawler (OAI-SearchBot, Claude-SearchBot or PerplexityBot) and 9.1% block a user-triggered agent (ChatGPT-User, Claude-User or Perplexity-User).
• The most-blocked user agents are CCBot (17.6%), Bytespider (17.4%), GPTBot (16.5%) and ClaudeBot (15.8%). The least blocked are OAI-SearchBot and Claude-SearchBot (5.5% each).
• Two in three GPTBot blockers keep OAI-SearchBot allowed. 484 of the 728 sites that block GPTBot (66.5%) leave OAI-SearchBot free to crawl.
• Full opt-outs are rare. Only 5.4% (240) block all five of GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot, and 2.2% (98) block all 16 AI user agents we tested.
• Server-level blocking is real. 6.1% (269 of 4,443) of sites that served our browser request refused an identical request whose user agent said OAI-SearchBot. In 4.6% (204) our own declared research crawler was allowed, which points to an AI-specific rule. Of the refusing sites that have a robots.txt, 72.3% (138 of 191) do not block OAI-SearchBot there.
• llms.txt has reached 14.0% of live websites (681 of 4,848) and 18.8% of live sites in the top 1,000. Another 16.1% answer /llms.txt with an HTML page instead of a text file.
• HTTPS is close to universal, but security headers are not. 97.7% of reachable domains answer over HTTPS. Only 57.6% of live HTTPS homepages send HSTS, 35.8% send a Content-Security-Policy, and 5.1% send all five headers we checked.
• Half of domains enforce DMARC. 50.4% of the 9,961 domains that resolve have DMARC at quarantine or reject, while 32.5% have no DMARC record at all.

Which AI crawlers are blocked most

We count a site as blocking a bot when its robots.txt disallows the root path / for that bot under the standard robots.txt rules (RFC 9309). For the "AI blocking" figures we also require that Googlebot is still allowed. Without that rule, the 118 sites (2.7%) that block every crawler, including Google, would be counted as making an AI decision.

A few points worth drawing out:

• Most site owners treat training and search differently. 567 sites (12.8%) block at least one training crawler but no AI search crawler. That matches the advice most vendors now give: you can opt out of model training and still be cited in AI answers. See our guide on whether to block or allow AI crawlers.
• Naming a bot is not the same as blocking it. 28.2% of robots.txt files (1,248) mention at least one AI user agent by name. GPTBot is named in 21.9% (971) but blocked from the whole site in 15.6% (689). The rest only restrict some paths or allow it explicitly.
• Counting sites that block everything raises the numbers a little. Including sites whose rules also block Googlebot, 19.0% of robots.txt files (841) disallow GPTBot.

Among the 300 most popular domains, the robots.txt files of nytimes.com, bbc.com and amazon.com disallow both GPTBot and OAI-SearchBot, while ebay.com, linkedin.com and medium.com disallow GPTBot but do not block OAI-SearchBot from the whole site. These are policy examples on the day we checked, not judgements.

The biggest sites block more, especially AI search

The gap is widest for AI search crawlers: the top 1,000 are almost twice as likely to block one. We did not measure why. One plausible reason is that many of the largest sites are publishers and platforms whose content is the product, while for most other businesses a citation in an AI answer is free visibility.

As a share of all live websites (not only those with a robots.txt), 16.0% (774 of 4,848) block at least one AI user agent and 11.7% (568) block GPTBot.

Blocking at the server, not just in robots.txt

robots.txt is a request, not a lock. Firewalls and CDNs can also refuse AI bots outright, and this does not show up in robots.txt at all. To measure it, we sent each homepage two requests that were identical except for the user agent: one from a standard desktop Chrome browser, one using OpenAI's published OAI-SearchBot user agent.

• 6.1% (269 of 4,443) of sites that served the browser request refused the OAI-SearchBot request with HTTP 401, 402, 403, 429 or 503. Most were 403 Forbidden (226). 27 sites answered 402 Payment Required, the status code used by pay-per-crawl schemes.
• 4.6% (204) refused OAI-SearchBot but let our own declared research crawler through, which points to an AI-specific rule. The other 1.5% (65) also refused our research crawler, which looks more like general bot filtering.
• robots.txt does not tell the whole story. Of the 191 refusing sites that have a robots.txt, 138 (72.3%) do not block OAI-SearchBot there. Going the other way, only 20.5% (53 of 259) of sites that block OAI-SearchBot in robots.txt also refused it at the server.

Read this figure with care. Our request claimed to be OAI-SearchBot but came from our own network, not from OpenAI's published IP ranges. Some CDNs refuse any request that claims to be a known bot from an unverified address, so part of this blocking may target impersonators rather than the real crawler. What it does show is that a meaningful share of popular sites now make AI decisions at the edge, out of sight of anyone reading their robots.txt.

The rate was similar in the top 1,000 (6.5%, 32 of 492) and in ranks 1,001 to 10,000 (6.0%, 237 of 3,951).

llms.txt: 14% of live sites, and many false positives

llms.txt is a proposed Markdown file at the root of a site that gives AI tools a short, curated map of its most useful pages. We counted a file as valid when /llms.txt returned HTTP 200, was not an HTML page, and either started with a Markdown heading or contained Markdown links.

Large developer and commerce platforms lead adoption. Cloudflare, GitHub, Stripe, Shopify, PayPal and NVIDIA all served a valid llms.txt on the day we checked. The median valid file was 7,869 bytes.

Two cautions:

• A 200 status is not proof of an llms.txt. More sites return an HTML page for /llms.txt (usually their homepage or a "page not found" template served with status 200) than publish a real file. Any audit that only checks the status code will overcount.
• Some sites send mixed signals. 73 of the 681 sites with a valid llms.txt also block at least one AI user agent in robots.txt. That can be deliberate (allow search, block training), but it is worth checking that the two files say what you mean.

We also found the newer Content-Signal line in 6.8% of robots.txt files (299). It is a machine-readable statement of how content may be used, popularised by Cloudflare's managed robots.txt. Where present, search=yes (298) and ai-train=no (242) were the most common values.

HTTPS and security headers

• HTTPS works on 97.7% of the 6,692 domains that answered over HTTP or HTTPS. 1.7% (111) had a certificate or TLS error.
• 85.8% of domains that answer on plain http:// redirect to HTTPS (5,025 of 5,856). The rest either serve the site over HTTP or redirect elsewhere.
• On the 4,839 live HTTPS homepages:

The top 1,000 do better: 69.5% send HSTS and 54.6% send a Content-Security-Policy, against 56.2% and 33.5% for ranks 1,001 to 10,000. Version disclosure is less common than it used to be. 79.7% of homepages send a Server header, but only 7.2% include a version number in it, and 14.2% send X-Powered-By.

SPF and DMARC

We checked public DNS records for the 9,961 domains that resolved. Eight no longer existed and 31 returned a DNS server failure. This includes infrastructure domains that never send email, so we also show figures for the 7,316 domains that publish MX records and therefore receive mail.

• Among domains that receive email, 60.9% enforce DMARC (4,453 of 7,316), and 17.3% (1,264) have no DMARC record at all.
• 58.2% of domains include a reporting address (rua=), so 9.2% publish DMARC without asking for the reports that make enforcement safe.
• SPF endings: 37.0% use -all, 32.8% use ~all, 1.6% use ?all and 3 domains use +all. 52 domains (0.5%) publish two SPF records, which makes SPF fail.
• CAA records appear on 20.6% of domains and DNSSEC (a DS record at the parent zone) on 10.4%.

For a deeper look at a single country, see our study of the 1,000 most popular .uk domains.

What this means for site owners

The sites in this study are among the most visited on the web and have the budgets to make deliberate choices. Smaller sites usually inherit their AI policy from a CMS plugin, a hosting default or a CDN switch. Use these numbers as a benchmark, then decide on purpose.

1. Decide training and search separately. Blocking GPTBot, ClaudeBot or Google-Extended limits use of your content for model training. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those AI search results. Most sites in this study that block one allow the other. Check what your robots.txt actually says for each bot with our AI Crawler Checker, and test individual rules with the robots.txt tester.
2. Check your CDN and firewall, not just robots.txt. Bot-management settings can refuse AI crawlers even when robots.txt allows them. If you want to appear in AI answers, make sure the setting matches your intent. Our post on Cloudflare's AI crawler controls covers the most common case.
3. See whether AI tools can read your content. The AI Content Visibility Checker shows how your pages look to AI systems, including content that only appears after JavaScript runs.
4. If you publish llms.txt, make it a real file. Serve it as plain text or Markdown with a 200 status. Make sure missing files return 404 rather than your homepage, or every tool will think you have one.
5. Finish HTTPS properly. Redirect http:// to https://, then add HSTS with a long max-age. Check your certificate and redirects with the SSL checker, and the rest of your headers with the Security Headers checker.
6. Enforce DMARC. Publish SPF, turn on DKIM, start DMARC at p=none with a rua= address, then move to quarantine and reject. The DMARC checker shows your current policy and reporting address.

Methodology

• Sample: the top 10,000 domains of the Tranco list ZJ8PG, generated on 28 October 2026. It aggregates rankings from CrUX, Farsight, Majestic, Cloudflare Radar and Cisco Umbrella for 29 September to 28 October 2026, filtered to registrable (pay-level) domains. Tranco is a ranking designed for research: tranco-list.eu/list/ZJ8PG.
• Collection: all web requests were made on 29 October 2026 between 14:22 and 16:03 UTC from a single machine, with up to 50 domains in parallel, a 10-second timeout and redirects followed (up to 5). 1,134 domains that failed with a timeout or connection error were retried once at lower concurrency, and the retry result was used. DNS records were queried the same day through Cloudflare's DNS-over-HTTPS resolver.
• Requests per domain: the HTTPS homepage (falling back to www. when the bare domain did not answer, which applied to 710 domains), /robots.txt, /llms.txt, the http:// homepage, and the homepage twice more for the user agent comparison. We did not crawl beyond these URLs, store page content or attempt anything beyond ordinary GET requests.
• User agents: MakeItFlowResearch/1.0 (+https://makeitflow.si/research) for all standard requests. For the server comparison only, a desktop Chrome 130 user agent string and OpenAI's published OAI-SearchBot string (Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot).
• Reachability: of the 10,000 domains, 6,692 answered over HTTP or HTTPS and 4,848 returned a successful (2xx) homepage over HTTPS. Of the 3,461 domains with no HTTPS response, 2,834 had no web address in DNS for either the bare domain or www.. Tranco ranks domains by all kinds of traffic, and many of these are API, CDN, advertising and tracking domains that do not host a website. The rest timed out (221), refused the connection (206), had a TLS error (173) or failed in other ways (27). Every unreachable domain is kept in the denominators where it applies, as shown with each figure.
• robots.txt: counted as valid when it returned HTTP 200 with plain-text content (4,425 domains). Responses that were HTML pages were treated as invalid. We parsed the files with our own implementation of RFC 9309: user-agent groups are matched on the exact product token, case-insensitively; groups for the same agent are merged; bots with no group of their own fall back to *; the longest matching rule wins, with Allow winning ties; and * and $ wildcards are supported. "Blocked" means the root path / is disallowed. We did not use Python's built-in urllib.robotparser, which applies the first matching rule rather than the longest and matches user agents by substring. 54 files let GPTBot fetch the homepage but block other paths (for example with Allow: /$), and these are not counted as blocking.
• Bot groups: training is GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent and cohere-ai. AI search is OAI-SearchBot, Claude-SearchBot and PerplexityBot. User-triggered is ChatGPT-User, Claude-User and Perplexity-User. Amazonbot is reported separately as mixed use. The "five major" crawlers are GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot.
• Server-level blocking: counted only when the browser request got a 2xx or 3xx status and the OAI-SearchBot request got 401, 402, 403, 429 or 503. Sites that did not give the browser request a 2xx or 3xx response, including 1,402 that refused it with 401, 402, 403, 429 or 503, are excluded from this measure.
• Security headers were read from the final homepage response (after redirects) to our research user agent. A site may send different headers to a real browser or on other pages.
• Limitations: this is a one-day snapshot. robots.txt rules apply per host, so a subdomain may have different rules. We checked whether bots are blocked, not whether they comply. The server-level test used a claimed user agent from a non-vendor network (see above). For DNS, we checked SPF, DMARC, MX, CAA and DS at the registrable domain only. We did not check DKIM, which needs a selector, and a DS record shows that DNSSEC is configured, not that it validates correctly. 43 domains had at least one DNS query fail.

Download the data

The aggregate results, with every count and denominator in this article split by rank band, are available as a CSV: summary.csv. It contains no per-site data. Journalists and researchers can contact hello@makeitflow.si with questions about the method.

How to cite this study

Make It Flow (2026). AI Crawler Blocking Study 2026: We Checked the Top 10,000 Websites. Data collected 29 October 2026 from the Tranco top 10,000 (list ZJ8PG). https://makeitflow.si/blog/ai-crawler-blocking-study-top-10000-websites-2026

When quoting a single figure, please include its denominator (for example, "22.4% of the 4,425 top-10,000 sites with a robots.txt") and link to this page.

Frequently asked questions

What percentage of websites block AI crawlers?

Among the 10,000 most popular domains, 22.4% of the 4,425 sites with a valid robots.txt block at least one of the 16 AI user agents we tested while still allowing Googlebot. As a share of all 4,848 live websites, it is 16.0%. The top 1,000 sites block more often (27.6%).

What percentage of sites block GPTBot?

16.5% of sites with a robots.txt (728 of 4,425) disallow GPTBot from the whole site while allowing Googlebot. Among the top 1,000 it is 19.2%. Two in three of these sites still allow OAI-SearchBot, which powers ChatGPT search.

Does blocking GPTBot remove my site from ChatGPT search?

Not on its own. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and pages opened at a user's request use ChatGPT-User. Each has its own robots.txt token, so you can block one and allow the others.

How many websites have an llms.txt file?

In our study, 14.0% of live websites in the Tranco top 10,000 (681 of 4,848) served a valid llms.txt, and 18.8% of live sites in the top 1,000. Another 16.1% returned an HTML page at /llms.txt, so checking the status code alone overcounts adoption.

Is robots.txt enough to block AI crawlers?

robots.txt is a public request that well-behaved crawlers follow. It does not technically stop anything. 6.1% of sites in our study also refused a request identifying as OAI-SearchBot at the server, usually through a firewall or CDN rule. If you want to block or allow AI crawlers reliably, check both your robots.txt and your CDN or firewall settings.

How can I check my own website?

Use the AI Crawler Checker to see which AI bots your robots.txt allows, the AI Content Visibility Checker to see what AI tools can read, and the Security Headers, SSL and DMARC checkers for the rest of this study's checks.