Published studies
- AI crawler blocking across the top 10,000 websites (October 2026): robots.txt rules for 16 AI user agents, server-level blocking, llms.txt, HTTPS, security headers and DMARC. Aggregate data: summary.csv.
- Email security across UK domains (2026): SPF, DKIM and DMARC adoption.
You may quote and chart our findings with a link to the study page. Please quote each figure with its denominator, as given in the study.
About MakeItFlowResearch
If you found MakeItFlowResearch/1.0 (+https://makeitflow.si/research) in your server logs, it was us
collecting data for one of the studies above. It is a one-off, low-volume check, not an ongoing crawl.
- What it requests: at most a handful of public URLs per domain, such as
/robots.txt,/llms.txtand the home page, plus a plain http:// request to check the redirect to https. It does not follow links, submit forms or log in. - What it keeps: status codes, response headers relevant to the study, and the text of robots.txt and llms.txt. We publish aggregate figures only and do not single out small sites.
- Public DNS: we also read public DNS records (SPF, DMARC, MX, CAA) through a public resolver, as any mail server or browser does.
Opting out
Add a group for MakeItFlowResearch to your robots.txt with Disallow: /, or email
hello@makeitflow.si with your domain and we will exclude it from future studies.