Noindex and robots.txt Disallow both sound like ways to keep a page out of Google, so people use them interchangeably. They aren't interchangeable, and combining them is one of the most common reasons an unwanted page stays in search results for months. This guide covers the difference between noindex and robots.txt Disallow, why blocking a page can keep it indexed, how to noindex PDFs with X-Robots-Tag, and the right way to remove a page from Google.
Key takeaways
- Robots.txt Disallow controls crawling while noindex controls indexing, so a disallowed page can still appear in Google if others link to it.
- Never combine Disallow and noindex on a page you want removed, because Google cannot crawl the page to see the noindex rule.
- Use an X-Robots-Tag noindex HTTP header for PDFs and other files that cannot carry a robots meta tag.
- Protect staging sites and sensitive content with a password, as neither robots.txt nor noindex is a security control.
The short answer: crawling vs indexing
The two controls work at different stages:
robots.txt Disallow |
noindex |
|
|---|---|---|
| What it controls | Crawling: whether Googlebot may fetch the URL | Indexing: whether the URL may appear in search results |
| Where it lives | One file at /robots.txt |
A meta tag in the page's HTML, or an X-Robots-Tag HTTP header |
| Can the URL still appear in Google? | Yes, if other pages link to it | No, once Google has recrawled the page and seen the rule |
| Works for PDFs and images? | Blocks crawling of any file type | Yes, using the X-Robots-Tag header |
| Main use | Managing crawl load, keeping crawlers out of low-value areas | Keeping specific pages out of search results |
| Security control? | No: the file is public | No: the page is still publicly reachable |
Google's robots.txt documentation is explicit: robots.txt is "used mainly to avoid overloading your site with requests" and "it is not a mechanism for keeping a web page out of Google."
Rule of thumb: use Disallow to control crawling, noindex to control indexing, and a password when the content must stay private.
Why a page blocked in robots.txt can stay indexed
This surprises a lot of site owners. Google says it "might still find and index a disallowed URL if it is linked from other places on the web." In that case the URL can appear in results, usually without a description, because Google was never allowed to read the page.
In Search Console's Page indexing report, these URLs show as "Indexed, though blocked by robots.txt".
The robots.txt plus noindex trap
The worst combination is adding both:
# robots.txt
User-agent: *
Disallow: /old-offers/
<!-- on /old-offers/summer.html -->
<meta name="robots" content="noindex">
It looks thorough. It isn't. Google's noindex documentation warns that if a page is blocked by robots.txt, "the crawler will never see the noindex rule". The robots meta tag specification adds that indexing rules on a blocked page "will not be found and will therefore be ignored." The Disallow stops Google reading the noindex, so the page can stay indexed indefinitely.
The fix is counterintuitive: remove the Disallow so Google can crawl the page and see the noindex.
What about "Noindex:" inside robots.txt?
Google stopped supporting noindex rules in robots.txt in September 2019, and they were never an official part of the standard. If you still have lines like Noindex: /private/ in your file, they do nothing in Google. Our robots.txt guide covers the syntax that is supported.
How to add noindex correctly
The robots meta tag (HTML pages)
Put this in the <head> of the page:
<meta name="robots" content="noindex">
To target only Google, use name="googlebot". You can combine rules, for example noindex, nofollow, though noindex on its own is usually what you want. Links on a noindexed page can still help Google discover other pages.
Platform settings
You rarely need to edit code by hand:
- WordPress: SEO plugins such as Yoast SEO, Rank Math and All in One SEO have a per-page setting to exclude the page from search results. Settings > Reading > Discourage search engines applies noindex across the whole site. It's useful on staging, and a disaster if it's left on after launch.
- Shopify: Shopify's help centre shows how to add a conditional noindex tag to
theme.liquidfor specific page handles. Shopify also describes aseo.hiddenmetafield that adds noindex and removes the page from the sitemap. - Wix: In a page's SEO settings, turn off the option that lets search engines index the page.
- Squarespace: Page settings include an SEO option to hide the page from search results.
X-Robots-Tag for PDFs and other files
A PDF has no <head>, so a meta tag isn't possible. Send the rule as an HTTP response header instead. Google supports noindex and the other robots rules in the X-Robots-Tag header for any file type.
Apache (.htaccess or virtual host, cPanel hosting):
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex"
</Files>
This needs mod_headers, which almost every host enables.
Nginx:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex" always;
}
Cloudflare: Create a Response Header Transform Rule that matches the file path or extension (for example, URI path ends with .pdf) and sets X-Robots-Tag to noindex. This works whatever platform sits behind Cloudflare.
X-Robots-Tag is also useful for noindexing a whole directory or file type in one rule, such as internal search results or print versions.
To confirm the header is actually being sent:
curl -I https://www.example.com/files/price-list-2024.pdf
Look for X-Robots-Tag: noindex in the response. Our free meta tag checker (free account required) also flags noindex in either the robots meta tag or the X-Robots-Tag header, alongside the title, meta description and canonical.
When to use each
| Situation | Use | Why |
|---|---|---|
| Thank-you and order confirmation pages | noindex |
Keep them out of results; crawling them is harmless |
| Internal site search results | noindex, optionally Disallow later |
Search result pages are thin, near-endless and rarely useful as landing pages |
| Faceted or filter URLs creating thousands of combinations | robots.txt Disallow | The problem is crawl waste, not one page showing in results |
| Old PDFs you want out of results | X-Robots-Tag: noindex |
PDFs can't carry meta tags |
| Staging or development site | Password protection (HTTP authentication) | Robots rules don't stop people or rogue crawlers |
Admin areas (/wp-admin/) |
Disallow plus authentication | Saves crawl effort; login protects it |
| Duplicate versions of a page | rel="canonical" |
Consolidates signals; noindex would discard them. See our canonical tags guide |
| Content that's gone for good | 404 or 410 status | The cleanest signal that the page no longer exists |
| Images you don't want in Google Images | robots.txt Disallow for the image path | Google says robots.txt can keep image, video and audio files out of results |
Never block CSS or JavaScript files that your pages need to render. Google needs them to understand the page.
How to remove a page from Google properly
Follow these steps in order.
- Decide whether the page should exist. If not, delete it and return a
404or410. If it moved, use a301redirect to the closest equivalent. - If the page must stay live but out of search, add
noindexusing the meta tag, platform setting orX-Robots-Tagheader. - Remove any robots.txt Disallow covering that URL so Google can recrawl it and see the change.
- Remove the URL from your XML sitemap. A sitemap that lists noindexed URLs sends mixed signals. Our sitemap checker shows what's listed.
- Ask Google to recrawl. Use URL Inspection in Search Console and select Request indexing. Google warns it "may take months for Googlebot to revisit a page" otherwise.
- For urgent cases, use the Removals tool too. A temporary removal hides the URL from Google results for about six months. It doesn't stop crawling or remove the page from the web, so steps 1 to 5 must be done as well, or the page can come back when the block expires.
- Check the result. In the Page indexing report, the URL should move to "Excluded by 'noindex' tag" or "Not found (404)".
Can I add the Disallow back afterwards?
Once a page has dropped out of the index, you can disallow it to save crawl effort. Be aware that if other sites link to it, Google may list the bare URL again, because it can no longer see the noindex. For most small sites, leaving noindexed pages crawlable is the safer choice. Crawl budget is rarely a concern for small sites.
If the content is sensitive
If you've accidentally published personal data, prices or documents that should never have been public, robots rules aren't enough. Put the content behind a password or remove it, then use the Removals tool. Google lists password protection and deletion as the reliable permanent options. Noindex is "less secure" because the page is still reachable by anyone with the link.
Common mistakes to check for
- A site-wide
noindexleft on after a redesign or launch. Our website launch checklist includes this check. - A Disallow and a noindex on the same URL.
Noindex:lines in robots.txt that Google ignores.- A canonical tag pointing to a page that is itself noindexed.
- PDFs left indexed because nobody added an
X-Robots-Tagheader. - A noindexed page still listed in the XML sitemap.
Use our robots.txt generator to build a clean file in the browser, with your sitemap line and optional AI crawler rules. To spot stray noindex tags, robots.txt problems and sitemap issues across your site in one scan, run a free website audit.
Frequently asked questions
Does robots.txt Disallow remove a page from Google?
No. Disallow stops Google crawling the page, but the URL can still be indexed and shown without a description if other pages link to it. Use noindex, a 404 or 410 status, or password protection to remove it.
Should I use noindex and Disallow together?
Not while you want the page removed. If robots.txt blocks the page, Google can't crawl it and never sees the noindex, so the page can stay indexed. Remove the Disallow until Google has processed the noindex.
How do I noindex a PDF?
Send an X-Robots-Tag: noindex HTTP header with the file, using your Apache or Nginx configuration, a Cloudflare Transform Rule or your host's header settings. PDFs can't carry a robots meta tag.
How long does noindex take to work?
It depends on when Google next crawls the page, which can be days for popular pages and months for rarely visited ones. Request indexing through URL Inspection in Search Console to speed it up, and use the Removals tool for urgent cases.
What does "Indexed, though blocked by robots.txt" mean?
Google has indexed the URL, usually because other pages link to it, but robots.txt stops it reading the content. If you want the page out of Google, remove the Disallow and add noindex. If you want it indexed, remove the Disallow so Google can read it.
Related reading
- Google Updates Its User-Generated Spam Guidance: What Sites Should Change: Google updated its user-generated spam guidance on October 26, 2026: keep share links and new users' pages out of the index and watch Search Console for abuse.
Sources
- Google Search Central: Introduction to robots.txt
- Google Search Central: Block search indexing with noindex
- Google Search Central: Robots meta tag, data-nosnippet and X-Robots-Tag specifications
- Search Console Help: Removals tool
- Google Search Central Blog: A note on unsupported rules in robots.txt (July 2019)
- Shopify Help Center: Hiding a page from search engines
Written by the MakeItFlow team
We build the website audit these guides are based on. Every fact is checked against the primary sources listed above and this guide was last reviewed on 27 October 2026 . How we research and check
Check your own site in 30 seconds
Run a free audit covering speed, SEO, DNS, email authentication and security headers — and get a prioritised fix list.
