AI Crawlers Overloading Your Website? How to Stop Them Fast

If an AI crawler is overloading your website, first find its user agent in your access logs, then block or throttle it at the server or firewall, which takes effect immediately. After that, add a robots.txt rule for the long term, because crawlers can cache robots.txt for up to 24 hours. Never block Googlebot or Bingbot by mistake.

This became a practical problem for many small sites in October 2026. Below is what happened, how to tell a real crawler from a fake one, and the exact rules to use, from emergency relief to a sensible permanent policy.

What small sites reported in October 2026

On November 2, 2026, Fast Company reported on several small business owners whose sites were swamped by Meta's crawlers:

• An artist's online shop. Meta-ExternalAgent got stuck in a loop on the site's "remove from cart" function, making more than 100 requests a minute and roughly 550,000 hits over a few days. CPU reached 80% and the host warned the site could go down. After she blocked that bot, Meta-WebIndexer pushed CPU to 100% and crashed the site. The flood only stopped once her host gave her .htaccess code.
• A one-person design business. Between September 30 and October 28, her server delivered 2.45 TB, and 97.6% of it went to Meta's crawlers, including 94,154 automatic downloads of zip files and PDFs. Earlier, in August, she paid her host $10.51 for exceeding a 2 TB plan limit.
• A community history project had to double its server size after Meta's and Alibaba's crawlers nearly crashed it.

Two patterns stand out. First, blocking one bot often caused a sibling bot to surge, so each one needed its own rule. Second, robots.txt alone did not give immediate relief. Meta told Fast Company that "website managers can decide how our crawlers interact with their sites through tools like robots.txt."

The same article cites Cloudflare data showing bots made up 55.1% of US web traffic over the previous four weeks.

Know which bot does what

Blocking the right crawler matters, because some of them are what get you cited in AI answers. Here are the ones relevant to most small sites, based on each company's own documentation:

Sources: Meta's web crawler documentation, OpenAI's crawler overview and Anthropic's crawler help article. For the full list and a default policy, see our guide on whether to block or allow AI crawlers.

Be careful with facebookexternalhit. If you block it, links to your site shared on Facebook, Instagram or WhatsApp may show without a title or image.

Step 1: Confirm it really is a bot problem

Before changing anything, look at the evidence.

1. Open your raw access logs. In cPanel this is usually under Metrics, then Raw Access. On a VPS, check /var/log/apache2/access.log or /var/log/nginx/access.log.
2. Count requests by user agent. On a Linux server, a quick count looks like this:

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

3. Look at which URLs are hit. Cart, search, filter and calendar URLs can produce endless combinations. The artist's site above was hit on a "remove from cart" action, which is exactly this kind of trap.
4. Check bandwidth, not just hits. Large downloads (PDFs, zip files, videos) cost far more than page views.

Step 2: Verify the bot is who it says it is

Anyone can put "Googlebot" or "meta-externalagent" in a user agent string. Google's Gary Illyes said on a recent podcast, as reported by Search Engine Roundtable on October 30, 2026, that AI has made bot impersonation worse: "I think it is AI." Impersonators pretend to be Googlebot because "you can say you are Googlebot and they will let you in."

Google's verification guide gives two methods:

1. Reverse DNS. Run a reverse lookup on the IP from your logs (for example host 66.249.66.1). A real Googlebot resolves to a name ending in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that name and confirm it returns the same IP.
2. IP ranges. Match the IP against Google's published list, such as common-crawlers.json.

OpenAI publishes IP lists for each of its bots, for example openai.com/gptbot.json and openai.com/searchbot.json. Meta's documentation says IP-based allow-listing is "more secure" than relying on user agent strings.

If a "Googlebot" request fails verification, it is not Google, and blocking it will not hurt your search visibility.

Step 3: Stop an active flood immediately

robots.txt is a request, and crawlers re-read it on their own schedule. Meta says its crawlers may cache robots.txt for up to 24 hours, and OpenAI gives the same ~24 hour figure for its search crawler. When your site is going down now, use a server rule.

Apache or LiteSpeed (.htaccess)

Add this near the top of your .htaccess file, above WordPress or other rewrite rules. It returns 403 Forbidden to the named bots:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (meta-externalagent|meta-webindexer) [NC]
RewriteRule .* - [F,L]
</IfModule>

Edit the list in brackets to match what you found in your logs, separating names with |. Back up the file first. A typo in .htaccess can take the whole site down with a 500 error. Our .htaccess guide covers safe editing, and the .htaccess generator helps with redirect rules.

Nginx

Inside the relevant server block:

if ($http_user_agent ~* "(meta-externalagent|meta-webindexer)") {
    return 403;
}

Reload Nginx after testing the configuration with nginx -t.

Through your CDN or firewall

If you use a CDN with a web application firewall, a user agent rule there stops the traffic before it reaches your server, which saves bandwidth as well as CPU. Our Cloudflare AI crawler controls guide walks through the options, and what a WAF does explains the basics.

Step 4: Throttle instead of block, if you want to stay visible

Blocking a search crawler such as OAI-SearchBot or Claude-SearchBot can remove you from that assistant's answers. If the problem is speed rather than access, slow the bot down.

• Return 429 or 503 with Retry-After. On October 6, 2026, Google updated its crawl rate documentation to show how to add a Retry-After header to a 503 or 429 response, either as seconds (Retry-After: 120) or as a date and time. Google lowers its crawl rate for the whole hostname when it sees many 500, 503 or 429 responses.
• Keep it short. Google warns that serving these codes for more than one or two days can cause URLs to be dropped from its index. Use errors as an emergency brake, not a permanent setting, especially for Googlebot.
• Use Crawl-delay where it is supported. Anthropic says it supports the non-standard Crawl-delay directive, for example Crawl-delay: 1 under User-agent: ClaudeBot.
• Fix crawler traps. Add rel="nofollow" to cart and add-to-wishlist links, make cart actions use POST forms instead of plain links, and disallow endless parameter URLs in robots.txt.

Check that your server actually sends the status and header you expect with the free HTTP headers checker. The HTTP status codes guide explains how search engines treat each code.

Step 5: Set a permanent robots.txt policy

Once the emergency is over, write down your policy so it survives the next bot surge. A common setup for a small business that wants to appear in AI search but limit training and heavy crawling:

User-agent: meta-externalagent
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Crawl-delay: 5

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?add-to-cart=

Adjust it to your own goals. Some businesses want training crawlers too; others block everything except Google and Bing. Build the file with the robots.txt generator, test individual URLs with the robots.txt tester, and see which AI bots your current file allows with the AI crawler checker. The tools need a free account.

Keep the server rule from Step 3 in place for any bot that ignored robots.txt or kept surging after you blocked it.

Step 6: Monitor so you catch the next one early

1. Turn on your host's resource alerts for CPU, memory and bandwidth.
2. Review the top user agents monthly with the log command above.
3. Watch uptime from outside. Our website monitoring guide covers free options.
4. Ask your host for help early. In the Fast Company report, the fix that finally worked came from the host.

Frequently asked questions

How do I stop Meta AI crawlers from crawling my website?

Add User-agent: meta-externalagent with Disallow: / to robots.txt, and do the same for meta-webindexer if you don't want Meta AI search indexing. Meta says robots.txt changes can take up to 24 hours, so add a server rule that returns 403 for those user agents if the traffic is causing problems now.

Does robots.txt stop AI crawlers immediately?

No. robots.txt only works when a crawler re-reads it, and Meta and OpenAI both say that can take up to about 24 hours. Some user-triggered fetchers, such as Meta-ExternalFetcher, may bypass robots.txt entirely. For immediate relief, block at the server, CDN or firewall.

Will blocking AI crawlers hurt my Google rankings?

Blocking AI-specific crawlers such as GPTBot, ClaudeBot or meta-externalagent does not affect Google Search. Blocking Googlebot does remove you from Google Search, so always verify a bot claiming to be Googlebot before you block it.

How can I tell if Googlebot is fake?

Run a reverse DNS lookup on the IP address. Google says a real Googlebot resolves to a domain ending in googlebot.com, google.com or googleusercontent.com, and a forward lookup on that name returns the same IP. You can also check the IP against Google's published crawler IP ranges.

Can I slow down a crawler instead of blocking it?

Yes. Return a 429 or 503 status with a Retry-After header during heavy load, which Google documents for its crawlers. Anthropic supports the Crawl-delay directive for ClaudeBot. Keep error responses short-term, since Google warns that serving them for several days can drop URLs from its index.

Why did blocking one bot make another one worse?

Fast Company reported that after site owners blocked Meta-ExternalAgent, Meta-WebIndexer traffic surged on the same sites. Companies run separate crawlers for different purposes, and each follows its own rules. List every bot from the same company in your logs and decide on each one individually.

Sources

• Fast Company: Meta's web crawlers are making life hell for owners of small business websites (November 2, 2026)
• Meta for Developers: Meta web crawlers
• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Google: Reduce the Google crawl rate
• Search Engine Roundtable: Google crawl rate documentation adds Retry-After details (October 6, 2026)
• Google: Verifying Googlebot and other Google crawlers
• Search Engine Roundtable: Google says AI made bot impersonation worse (October 30, 2026)