Test robots.txt Rules Before They Cost You Traffic

To test a robots.txt rule, check a list of real URLs against the exact file, for each crawler that matters. Work out which group applies to that crawler, then which rule is the longest match. Do this before you publish, then confirm afterwards in Search Console. Most robots.txt disasters come from a rule that matches more than the author meant it to.

For a quick check of your live file, enter any page URL in our free robots.txt Tester. It fetches your robots.txt and shows whether Googlebot, Bingbot and the main AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot and DuckDuckBot) may fetch that path. This guide is about the logic behind those answers and a testing routine that catches mistakes before deployment. For syntax basics and copy-paste templates, see our robots.txt guide.

Why robots.txt mistakes are so expensive

A robots.txt file is short, and one character can block a whole site. Disallow: / instead of Disallow: (empty) blocks everything. A staging file deployed to production can remove a site from search over the following days. Things that make it worse:

• Caching. Google generally caches robots.txt for up to 24 hours, according to its robots.txt specification. A fix you deploy now may not be seen until tomorrow.
• Silent failure. Nothing on the website looks wrong. Pages still load for visitors. The first sign is often a traffic drop, or "Blocked by robots.txt" piling up in Search Console.
• It doesn't remove pages. Blocked URLs can stay indexed without a description. Robots.txt controls crawling, not indexing. Our guide on noindex vs robots.txt Disallow explains the difference.

How matching actually works (RFC 9309)

The Robots Exclusion Protocol is standardised as RFC 9309. Google documents its behaviour against it, and our tester implements the same rules. Testing comes down to two questions, answered in order.

Question 1: Which group applies to this crawler?

A group is one or more User-agent lines followed by rules. A crawler:

1. Looks for groups whose User-agent matches its product token (case-insensitive, so googlebot and Googlebot are the same).
2. If several groups name it, their rules are combined into one group.
3. Only if no group names it does it use the User-agent: * group.

The third point causes most of the mistakes. A specific group replaces the * group for that crawler. It doesn't add to it:

User-agent: *
Disallow: /checkout/
Disallow: /account/

User-agent: GPTBot
Disallow: /blog/

GPTBot is now blocked from /blog/ but allowed into /checkout/ and /account/, because it ignores the * group completely. If you want shared rules to apply to a named crawler, repeat them in its group.

Question 2: Which rule wins within that group?

• The rule with the longest matching path (most characters) wins.
• If an Allow and a Disallow match with equal length, Allow wins. RFC 9309 says the allow rule "SHOULD be used", and Google uses the least restrictive rule.
• If no rule matches, the URL is allowed.
• /robots.txt itself is always allowed.

Rule order in the file doesn't matter. Length does.

Wildcards

Only two special characters exist:

• * matches any sequence of characters, including none.
• $ at the end anchors the pattern to the end of the URL.

Paths are case-sensitive and matched from the start of the path, including any query string. There's no regex, no ? wildcard and no character classes.

A test table you can reuse

The quickest way to find out whether you really understand a rule is to predict the result, then check it. Here are cases that catch people out:

The .css row matters on WordPress. Rules like Allow: /*.css often fail to override a longer Disallow, which leaves Google unable to load stylesheets and render the page properly. If you need to block a folder but allow its assets, write the Allow with a longer path, such as Allow: /wp-content/themes/.

AI crawlers: one group per decision

With AI crawlers, the replacement rule from Question 1 can work for you or against you. Each one obeys its own group if it has one:

User-agent: *
Disallow: /checkout/
Disallow: /account/

# Model training: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# AI search: allow, but keep private areas out
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

What to test in a file like this:

• Each named crawler's path list is complete. The AI search group above repeats /checkout/ and /account/, because it doesn't inherit them from *.
• Google-Extended doesn't touch Search. Google says Google-Extended has no separate user-agent string in HTTP requests. Crawling is done by Google's normal crawlers, and the token only controls whether your content is used for Gemini model training and grounding. It doesn't affect inclusion or ranking in Google Search. So "Google-Extended: blocked" alongside "Googlebot: allowed" is expected, not a conflict.
• Googlebot hasn't been caught by accident. If you add a User-agent: Googlebot group for one exception, Googlebot stops reading the * group. Test Googlebot separately after any change like that.
• Product tokens are spelt correctly. Matching is case-insensitive, but Claude-Bot or GPT-Bot won't match anything.

To choose which AI crawlers to allow in the first place, our guide on whether to block or allow AI crawlers explains what each one is for. Our AI Crawler Checker shows your current policy at a glance. If you'd rather build the file from tick boxes, the Robots.txt Generator writes the groups for you.

The tools, and what each is good for

Search Console robots.txt report

The robots.txt report replaced the old robots.txt Tester in late 2023. It shows the robots.txt files Google found for your top 20 hosts, when each was last fetched, the fetch status, and any parse warnings or errors. After fixing a mistake, use Request a recrawl from the file's menu so Google picks up the new version sooner. It doesn't test individual URLs. Use URL Inspection for that, which reports "Blocked by robots.txt" when a rule applies. Our Google Search Console beginner's guide covers both reports.

Testing a draft before you deploy

Our tester and the Search Console report both read the live file. To test a draft, use one of these:

• Bing Webmaster Tools has a robots.txt tester with an editor, so you can paste a changed version and test URLs against it before uploading. See our Bing Webmaster Tools setup guide.
• Google's parser is open source at github.com/google/robotstxt. Once built, it tests one file, one user agent and one URL per run:

robots_main draft-robots.txt Googlebot https://www.example.com/shop/products/red-mug

• A staging site. Upload the draft to staging and point our tester at a staging URL, as long as staging is publicly reachable.

Be careful with general-purpose libraries. Python's built-in urllib.robotparser, for example, applies the first matching rule in file order rather than the longest, and doesn't support * wildcards inside paths. It can give different answers from Google for exactly the edge cases in the table above.

A testing routine for every robots.txt change

1. Keep a list of 15-30 critical URLs: home page, top landing pages, a product, a category, a blog post, a CSS file, a JS file, an image, plus URLs that should be blocked (checkout, account, internal search, filter parameters).
2. Write down the expected result for Googlebot, Bingbot and each AI crawler you name in the file.
3. Test the draft with Bing's editor or Google's parser, and fix anything that doesn't match what you expected.
4. Deploy, then check the live file: open https://yourdomain/robots.txt and confirm it returns 200 with the new content. A 5xx on robots.txt makes Google stop crawling for up to 12 hours (see our HTTP status codes and SEO guide).
5. Run the critical URLs through our robots.txt Tester.
6. Request a recrawl in the Search Console robots.txt report, and spot-check two or three URLs with URL Inspection.
7. Watch the Pages report over the next one to two weeks for a rise in "Blocked by robots.txt".

If your site is built by a deployment pipeline, automate steps 3 and 4. Run Google's parser against the URL list in CI, and fail the build if the production robots.txt contains Disallow: / under User-agent: *. That one check prevents the most common disaster: staging rules going live.

FAQ

How do I test if a URL is blocked by robots.txt?

Enter the URL into a robots.txt tester that applies RFC 9309 matching, such as our free tool, which shows the result for Googlebot, Bingbot and the main AI crawlers. For Google specifically, use URL Inspection in Search Console, which reports "Blocked by robots.txt" if a rule applies.

Which rule wins when Allow and Disallow both match?

The rule with the longest matching path wins, whatever order the rules are in. If both rules are the same length, Allow wins. So Allow: /shop/products/ beats Disallow: /shop/ for product pages, but Allow: /*.css loses to the longer Disallow: /wp-content/.

Does Googlebot follow the User-agent: * rules if it has its own group?

No. A crawler uses only the most specific group that names it, and ignores the * group completely. If you add a Googlebot group, copy any shared Disallow rules into it as well, or Googlebot will be allowed into areas you meant to block.

What happened to Google's robots.txt Tester?

Google retired the old robots.txt Tester in late 2023 and replaced it with the robots.txt report in Search Console. The report shows the files Google has fetched, their status and any parse errors, and lets you request a recrawl. To test individual URLs, use URL Inspection, Bing's tester or Google's open-source parser.

Does blocking Google-Extended remove my site from Google Search?

No. Google-Extended is a robots.txt token, not a separate crawler, and Google says it doesn't affect inclusion or ranking in Google Search. It only controls whether your content is used for Gemini model training and grounding.

How long does a robots.txt change take to work?

Google generally caches robots.txt for up to 24 hours, so a change usually takes effect within a day. Requesting a recrawl in the Search Console robots.txt report can speed this up. Other crawlers, including AI bots, follow their own refresh schedules.