A single wrong line in robots.txt can stop Google from crawling your whole site, and the file is easy to forget once it ships. This guide covers how the file works, how Google settles conflicting rules, examples you can copy, and the mistakes we see most often in real audits.
Key takeaways
- Robots.txt controls crawling, not indexing, so a blocked URL can still appear in Google if other pages link to it.
- Googlebot follows only its most specific matching user-agent group, so repeat any shared rules in every named group.
- Never block CSS or JavaScript files, because Google renders pages like a browser and may misjudge a page it cannot fully load.
- A Disallow: / line copied from staging is the most damaging robots.txt mistake, so check the live file on launch day.
What robots.txt does (and what it doesn't)
robots.txt is a plain text file at the root of your host that tells crawlers which URL paths they may request. It controls crawling, not indexing. Mixing those two up causes most robots.txt problems.
Google's documentation is direct about this: robots.txt "is not a mechanism for keeping a web page out of Google." If other pages link to a blocked URL, Google can still index it. The result just won't have a description, because Google never fetched the page.
So:
- To save crawl activity on endless filter URLs, internal search results or staging paths, use robots.txt.
- To keep a page out of search results, use a
noindexmeta tag orX-Robots-Tagheader, or put the page behind a password.
Where the file must live
- It must be at the root:
https://www.example.com/robots.txt. Crawlers don't look for it in subdirectories. - It applies only to the exact protocol, host and port it's served from.
https://example.com/robots.txtdoesn't coverhttps://shop.example.com/orhttp://example.com/. Each subdomain needs its own file. - Google reads the first 500 KiB and ignores anything after that.
- It must be UTF-8 plain text.
Syntax basics
A robots.txt file is made of groups. Each group starts with one or more User-agent lines, followed by the rules for those crawlers.
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
Sitemap: https://www.example.com/sitemap.xml
Google supports four fields:
| Field | Purpose |
|---|---|
user-agent |
Which crawler the group applies to (* means all) |
disallow |
Path prefix the crawler should not request |
allow |
Path prefix that is allowed, used to override a broader disallow |
sitemap |
Absolute URL of an XML sitemap; can appear anywhere and more than once |
Google ignores other fields. That includes crawl-delay, which some other search engines still honor.
Wildcards
*matches zero or more of any character.$anchors the match to the end of the URL.
User-agent: *
# Block every URL with a session parameter
Disallow: /*?sessionid=
# Block PDFs, but not /pdf-guides/
Disallow: /*.pdf$
Case sensitivity
Field names and user-agent values are case-insensitive. Paths are case-sensitive, so Disallow: /Private/ doesn't block /private/.
How Google resolves conflicting rules
robots.txt is standardized as RFC 9309, and Google follows these rules:
- Only one group applies per crawler. Googlebot uses the most specific user-agent group that matches it. If you have a
User-agent: Googlebotgroup, Googlebot ignores theUser-agent: *group completely. It doesn't merge them. - The longest matching path wins.
Allow: /blog/press/beatsDisallow: /blog/for/blog/press/launchbecause it's more specific. - On a tie, the least restrictive rule wins. If
Allow: /pageandDisallow: /pageare equally long, Allow applies.
Rule 1 surprises people. This setup does not block Googlebot from /admin/:
User-agent: *
Disallow: /admin/
User-agent: Googlebot
Disallow: /tmp/
Googlebot matches its own group, so it only follows Disallow: /tmp/. Repeat any shared rules in every named group.
Copy-paste examples
Allow everything. An empty Disallow, or no robots.txt at all, means the same thing:
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
Block the whole site. Only use this on staging, and protect staging with a password as well:
User-agent: *
Disallow: /
A typical WordPress setup. This matches what WordPress serves by default:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/wp-sitemap.xml
Ecommerce faceted navigation. Stop crawlers from spending their time on sort and filter combinations:
User-agent: *
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?filter_
Disallow: /checkout/
Disallow: /account/
Keep images out of Google Images only:
User-agent: Googlebot-Image
Disallow: /private-images/
Opt out of AI training while staying in Search. Google-Extended is a robots.txt token that controls whether Google can use your content for its Gemini models. It has no effect on Google Search crawling or ranking.
User-agent: Google-Extended
Disallow: /
Common mistakes
1. Blocking CSS and JavaScript
Old SEO advice said to block /wp-includes/, /assets/ or /static/. Google renders pages much like a browser does. If it can't fetch your CSS and JS, it may see a broken or empty layout and misjudge mobile usability and content. Google's guidance is: if missing resources make the page harder to understand, don't block them.
# Don't do this
User-agent: *
Disallow: /wp-includes/
Disallow: /*.js$
Disallow: /*.css$
2. Using robots.txt to "noindex"
Google stopped supporting a noindex rule inside robots.txt in September 2019. It never worked reliably before that either. The real trap is worse: if you disallow a URL and add a noindex meta tag to it, Google can't crawl the page, so it never sees the noindex. The URL can stay indexed.
The fix is to remove the Disallow, add the noindex, and let Google recrawl:
<meta name="robots" content="noindex">
For PDFs and other non-HTML files, send the header instead:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex" always;
}
Once the URL has dropped out of the index, you can add the Disallow back if crawl load matters.
3. Shipping the staging file to production
Disallow: / copied from staging is the most damaging robots.txt mistake, and it happens on launch day more than any other time. Add a robots.txt check to your website launch checklist.
4. Returning the wrong status code
How Google treats the file depends on the HTTP status:
| Response | What Google does |
|---|---|
| 200 | Uses the rules |
| 4xx (except 429) | Acts as if there's no robots.txt and crawls everything |
| 5xx or 429 | Stops crawling for 12 hours, then uses the last cached copy for up to 30 days |
| 3xx | Follows up to five redirect hops, then treats the result as a 404 |
A misconfigured firewall or CDN that returns 503 for /robots.txt can quietly slow or stop crawling across the whole site.
5. Blocking pages you want consolidated
If a duplicate URL is blocked, Google can't see its canonical tag or follow its redirect. Leave duplicate URLs crawlable and handle them with canonical tags or redirects.
6. Forgetting the cache
Google usually caches robots.txt for up to 24 hours. Fixes aren't instant. After an urgent change, you can ask for a recrawl from the robots.txt report in Search Console.
Testing your file
- Open
https://yourdomain.com/robots.txtin a private window. Check that it returns 200 and plain text, not an HTML error page. - In Google Search Console, open Settings > robots.txt. The report shows which robots.txt files Google found, when it last crawled them, and any parsing errors or warnings.
- Use the URL Inspection tool on important pages. If one is blocked, the inspection result says "Blocked by robots.txt."
- To test rules before you deploy, Google's open-source robots.txt parser runs locally.
Frequently asked questions
Do I need a robots.txt file at all?
No. If the file is missing and returns a 404, Google crawls everything it can find. A short file is still worth having so you can point to your sitemap and keep crawlers out of low-value areas like cart or internal search pages.
Will blocking a page in robots.txt remove it from Google?
No. Blocking stops crawling, but the URL can still be indexed from links on other pages, just without a description. To remove a page, let Google crawl it and serve a noindex directive. For urgent cases, use the Removals tool in Search Console as well.
Does Google respect crawl-delay?
No. Google ignores crawl-delay. Googlebot slows down on its own when your server returns 500, 503 or 429 responses. Bing and some other crawlers do support crawl-delay.
Should I list my sitemap in robots.txt?
Yes. The Sitemap: line is supported by Google and Bing, needs an absolute URL, and can be repeated for several sitemaps. You should still submit the sitemap in Search Console and Bing Webmaster Tools so you get processing reports. See our XML sitemap guide.
What to do next
Open your live robots.txt and check three things. First, there's no stray Disallow: /. Second, nothing blocks CSS, JS or image folders. Third, no URL you've marked noindex is also disallowed. Then run a free audit to confirm that robots.txt returns 200, that it references your sitemap, and that it doesn't block your key pages. A whole-site crawl on a paid plan goes further and flags pages that are blocked, noindexed or canonicalized elsewhere.
Related reading
- Test robots.txt Rules Before They Cost You Traffic: A practical way to test robots.txt changes: how precedence and wildcards really match, per-crawler groups including AI bots, and which tools to trust.
Sources
Written by the MakeItFlow team
We build the website audit these guides are based on. Every fact is checked against the primary sources listed above . How we research and check
Check your own site in 30 seconds
Run a free audit covering speed, SEO, DNS, email authentication and security headers — and get a prioritised fix list.
