AI Crawler Blocking Study 2026: We Checked the Top 10,000 Websites
We checked the 10,000 most popular websites for AI crawler blocks, llms.txt, HTTPS, security headers, SPF and DMARC. 22.4% of sites with a robots.txt block at least one AI bot.
332 practical guides to website speed, SEO, email deliverability, DNS, security and automation, with fixes you can copy and paste.
We checked the 10,000 most popular websites for AI crawler blocks, llms.txt, HTTPS, security headers, SPF and DMARC. 22.4% of sites with a robots.txt block at least one AI bot.
Popular topics
Free website audit
Speed, SEO, DNS, email (SPF/DKIM/DMARC) and security in one scan, with a prioritised fix list.
Make It Flow assistant
Tools, guides and AI automation
Hi! I can find the right free tool, explain an audit result, or tell you how Make It Flow can automate any part of your business. What do you need?
Book a free 15-minute call
Pick a time that suits you. Times are shown in your time zone ().
A Make It Flow agent will confirm the call by email within 24 hours.
AI answers can be wrong. Chats are stored so the team can follow up. Privacy