AI Crawler Blocking Study 2026: We Checked the Top 10,000 Websites
We checked the 10,000 most popular websites for AI crawler blocks, llms.txt, HTTPS, security headers, SPF and DMARC. 22.4% of sites with a robots.txt block at least one AI bot.
332 practical guides to website speed, SEO, email deliverability, DNS, security and automation, with fixes you can copy and paste.
We checked the 10,000 most popular websites for AI crawler blocks, llms.txt, HTTPS, security headers, SPF and DMARC. 22.4% of sites with a robots.txt block at least one AI bot.
Emails landing in spam? Learn the four real causes: missing authentication, poor sender reputation, risky content and dirty lists, plus the fixes for each one.
Learn to read DMARC aggregate (rua) reports: what each XML field means, how to sort sources into legitimate, misconfigured and spoofed, and what to fix first.
Outlook enforces SPF, DKIM and DMARC for bulk senders and Gmail now rejects non-compliant mail. What email deliverability UK senders need in place now.
NCSC retired Mail Check on 31 March 2026 as phishing still hits 38% of UK businesses. How to keep DMARC monitored and protect email deliverability in the UK.
The current Gmail, Yahoo and Outlook.com sender rules: 5,000-a-day thresholds, SPF, DKIM, DMARC, one-click unsubscribe, 0.3% spam rate and enforcement dates.
A repeatable email security audit for MSPs and IT providers: check SPF, DKIM, DMARC, MTA-STS, TLS and blacklists on every client domain, then report clearly.
Email security for solicitors: how conveyancing payment fraud works, what the SRA and NCSC advise, and how SPF, DKIM, DMARC and staff checks protect clients.
Email fraud protection for accountants: how invoice fraud and fake HMRC emails target practices, and how DMARC, MFA and client verification keep money safe.
How to open full email headers in Gmail, Outlook and Apple Mail, read the Received chain bottom-up, check SPF, DKIM and DMARC, and spot spoofing or delays.
Automate DMARC aggregate report monitoring with free tools like parsedmarc, Postmark's digest or Cloudflare, set useful alerts and move from p=none to reject.
A one-page email authentication checklist for small businesses: SPF, DKIM, DMARC, unsubscribe, MTA-STS, BIMI and parked domains, with the records to publish.
Popular topics
Free website audit
Speed, SEO, DNS, email (SPF/DKIM/DMARC) and security in one scan, with a prioritised fix list.
Make It Flow assistant
Tools, guides and AI automation
Hi! I can find the right free tool, explain an audit result, or tell you how Make It Flow can automate any part of your business. What do you need?
Book a free 15-minute call
Pick a time that suits you. Times are shown in your time zone ().
A Make It Flow agent will confirm the call by email within 24 hours.
AI answers can be wrong. Chats are stored so the team can follow up. Privacy