robots.txt Tester

A robots.txt tester checks whether a specific page on your site may be crawled under the rules in your robots.txt file. Our tool fetches the live file from your domain, takes the exact path and query string you entered, and works out the answer for ten crawlers, including Googlebot, Bingbot and the main AI crawlers. You see whether the file loaded, which crawlers are allowed or blocked from that page, which of them have their own rules, and the file itself.

How to use the robots.txt tester

1. 1 Paste the page addressEnter the full URL of the page you want to test, such as https://example.co.uk/blog/post?ref=news. The path and query string are both tested.
2. 2 Sign in and run the testUse your free account to run it. We fetch robots.txt from the root of that host and read it as crawlers do.
3. 3 Check each crawlerThe table lists all ten crawlers as allowed or blocked, and marks those that have their own rules rather than falling back to the * group.
4. 4 Edit the file and retestNarrow or remove the rule that blocks a page you want crawled, upload the new robots.txt, then run the test again.

How robots.txt rules work

robots.txt is a plain text file at the root of a host, for example https://example.co.uk/robots.txt. The Robots Exclusion Protocol was used informally for decades and was finally standardised as RFC 9309 in 2022. Each host has its own file, so shop.example.co.uk and example.co.uk need a robots.txt each, and the http and https versions are treated separately too.

The file is made of groups. A group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules containing URL paths. Lines starting with # are comments, field names are not case sensitive, and the paths in rules are case sensitive, so Disallow: /Admin does not block /admin.

robots.txt controls crawling, not indexing. It asks well-behaved crawlers not to fetch certain URLs, but it is not access control: anyone can read the file and ignore it, so never rely on it to hide private content.

How the robots.txt tester decides allowed or blocked

Our tester follows the matching rules in RFC 9309, which Google and Bing also follow. First it picks the right group for each crawler. If a group names the crawler, it obeys that group only and ignores the * group entirely. If several groups name the same crawler, their rules are combined. Only when no group names it does the crawler fall back to the * group, and if there is no * group either, everything is allowed.

Next it compares the page path against the rules in that group. The rule with the longest matching path wins, whatever order the lines appear in. If an Allow and a Disallow rule match with paths of the same length, Allow wins. An empty Disallow line blocks nothing. In paths, * matches any run of characters and a $ at the end means the URL must end there.

Group names are matched against the crawler's user agent token, ignoring case. The tester looks for each crawler's exact token, so a group for Googlebot-Image does not apply to Googlebot in our results.

The crawlers we test

Every test checks the same path for ten crawlers, so you can see at a glance whether a rule meant for one is catching another. Googlebot and Bingbot are scored as pass or fail because blocking them stops a page ranking in search. The others are reported for information, as blocking them is often a deliberate choice.

• Search engines: Googlebot, Bingbot, Applebot and DuckDuckBot.
• OpenAI: GPTBot, which collects training data, and OAI-SearchBot, which fetches pages for ChatGPT search results.
• Anthropic: ClaudeBot for training data and Claude-SearchBot for search features.
• Others: PerplexityBot, and Google-Extended, a control token that tells Google whether your content may be used for its AI models without affecting Google Search.

When robots.txt does not load

The status code of the robots.txt file itself matters. A 200 response means the rules are read and applied. A 4xx response, including 404, means there is no file, and under RFC 9309 crawlers may fetch everything. Our tester reports this as information, not a fault.

A 5xx error or no response is different. Crawlers treat an unreachable robots.txt as a temporary instruction to crawl nothing, and Google pauses crawling for a period until it can read the file again. If the error persists for a long time, Google may eventually use a cached copy or crawl as normal. We flag this as a warning. Because there are no rules to read in that case, the crawler table shows pages as allowed, so act on the warning rather than the table.

We also read up to 512 KB of the file. Google ignores anything after the first 500 KiB, so if your file is near that size, important rules at the end may be missed.

robots.txt, noindex and crawl budget

If you want a page out of search results, a robots.txt block works against you. Google cannot crawl a blocked page to see a noindex tag, and it may still index the URL from links, showing it without a description. Allow crawling and use a meta robots noindex tag or an X-Robots-Tag header instead.

robots.txt is best used to keep crawlers away from URLs with no search value: internal search results, endless filter and sort combinations, basket and checkout steps, and staging areas linked by mistake. On large sites this saves crawl capacity for pages that matter. On a site of a few hundred pages, crawl budget is rarely a problem, and the safest robots.txt is a short one. Fixing weak internal linking and duplicate URLs at source does far more than adding Disallow rules.

Do not block CSS, JavaScript or image files that your pages need to render. Google renders pages much as a browser does, and blocked resources can make a page look broken or empty to it.

Sitemap lines and other directives

robots.txt can also point crawlers to your XML sitemaps with a Sitemap line containing a full absolute URL. These lines sit outside user agent groups and apply to every crawler. Open the file view in the results to confirm they point to the right, live address, particularly after a domain move or a switch to HTTPS. Other lines such as Crawl-delay do not change whether a page is allowed, so the tester does not use them in its answer.

Testing robots.txt changes safely

A single character in robots.txt can change which pages a search engine can reach, so treat edits with the same care as a code change. Before you upload a new file, list the URLs that matter most: the home page, top category and product pages, key service pages and recent articles. After uploading, run each of them through the tester and confirm that Googlebot and Bingbot are still allowed.

Keep a copy of the previous file, so you can restore it quickly if something goes wrong. If you use a CMS or SEO plugin that generates robots.txt automatically, check whether it overwrites a file you upload by hand, and whether a setting such as discourage search engines from indexing this site has been left switched on. Then watch the crawl stats and page indexing reports in Search Console and Bing Webmaster Tools over the following days for any sudden fall in pages crawled.

Examples you can copy

User-agent: *
Disallow:

Sitemap: https://example.co.uk/sitemap.xml

User-agent: *
Disallow: /search
Disallow: /basket
Disallow: /*?sort=

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /downloads/
Allow: /downloads/brochure.pdf

Common mistakes to avoid

Frequently asked questions