Robots.txt Generator

A robots.txt generator builds the small text file that tells search engines and AI crawlers which parts of your website they may fetch. Writing it by hand is easy to get wrong: one misplaced slash can hide a whole site from Google. Our generator runs entirely in your browser, starts from a WordPress or online shop preset if you want one, adds your sitemap, lets you opt out of AI training crawlers and gives you a file you can copy or download in seconds.

How to use the robots.txt generator

1. 1 Choose allow or blockPick Allow crawling for a normal public website. Block everything is only for staging or private sites, and adds Disallow: / for every crawler.
2. 2 Pick a preset and add pathsStart from Blank, WordPress or Online shop, then type any extra paths to keep crawlers out of, one per line, plus optional Allow exceptions.
3. 3 Decide on AI crawlersTick the AI crawlers you want to block, or use Block all of them. Each ticked bot gets its own group with Disallow: /.
4. 4 Add your sitemapPaste the full sitemap URL, including https://. Leave Crawl-delay empty unless a specific crawler is overloading your server.
5. 5 Copy or download and uploadCopy the file or download robots.txt, upload it to the root of your site, then open it in a browser to check it is live.

What a robots.txt file is and how this generator builds one

robots.txt is a plain text file at the root of a host, for example https://www.example.co.uk/robots.txt. It follows the Robots Exclusion Protocol, standardised as RFC 9309. The file is made of groups: each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules for paths. Crawlers that follow the standard read the file before fetching pages and obey the group that matches them most specifically.

Our robots.txt generator writes a User-agent: * group for all crawlers first. Allow exceptions are listed before Disallow paths, and if you add no paths it writes an empty Disallow: line, which means nothing is blocked. Each AI crawler you tick gets a separate group, and your sitemap goes at the end. Paths you type without a leading slash have one added, and you can separate them with new lines or commas.

Why a correct robots.txt matters for Google and AI search

For most small sites, robots.txt is less about saving crawl budget and more about avoiding disasters. A single Disallow: / under User-agent: * tells Googlebot and Bingbot to stay away completely, which removes you from normal results, Google AI Overviews and Microsoft Copilot answers. This happens surprisingly often after a site moves from a staging server to live.

A good file does three jobs: it keeps crawlers out of areas with no search value, such as admin screens, baskets and internal search results; it points every crawler at your XML sitemap; and it records your choice about AI training crawlers. Everything else should stay open, including the CSS, JavaScript and images Google needs to render your pages.

Using the WordPress and online shop presets

The WordPress preset blocks /wp-admin/ and keeps /wp-admin/admin-ajax.php open, because many themes and plugins call that file from public pages. This matches the rules WordPress itself outputs by default, so it is a safe starting point. Do not block /wp-content/ or /wp-includes/: they hold the stylesheets, scripts and images Google uses to render your pages.

The Online shop preset blocks /cart/, /checkout/ and /my-account/, plus URLs containing ?add-to-cart=, which WooCommerce creates for every add-to-basket link. These pages are personal to each visitor and have nothing to rank. If your shop uses different paths, edit the Disallow box to match what you see in your browser address bar.

• WordPress with Yoast SEO: paste the file into Yoast, Tools, File editor.
• WordPress with Rank Math: paste it into General Settings, Edit robots.txt.
• Shopify: the platform generates robots.txt; to change it, add a robots.txt.liquid template to your theme rather than uploading a file.
• Wix: use the robots.txt editor in the SEO settings.
• Squarespace: robots.txt is managed for you, and there is a setting to block known AI crawlers instead.

Blocking AI crawlers with a robots.txt generator

The AI section lists GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Gemini training), Applebot-Extended (Apple AI training), PerplexityBot and Bytespider (ByteDance). Ticking a box adds a group such as User-agent: GPTBot followed by Disallow: /. None of these affect Google or Bing search results.

Think carefully about PerplexityBot. Perplexity describes it as the crawler that finds and links sites in its answers, so blocking it is likely to stop Perplexity citing you. If you want to be cited by AI search tools while opting out of training, leave PerplexityBot unticked and do not block OAI-SearchBot or Claude-SearchBot, which are the search crawlers for ChatGPT and Claude. You can check the result afterwards with our AI Crawler Checker.

Wildcards and rule priority in a robots.txt file

Google, Bing and other major crawlers support two special characters in paths. An asterisk matches any run of characters, so Disallow: /*?sort= blocks every URL containing ?sort= wherever it appears. A dollar sign anchors the end of the URL, so Disallow: /*.pdf$ blocks addresses ending in .pdf but not /pdf-guides/. The shop preset uses the asterisk for its add-to-cart rule, and you can type your own patterns into the Disallow and Allow boxes.

When several rules match the same URL, the most specific one wins, which means the rule with the longest path. If an Allow and a Disallow rule are the same length, Allow wins. That is why the WordPress preset can block /wp-admin/ while keeping /wp-admin/admin-ajax.php open: the longer Allow rule takes priority for that one file. The order of lines inside a group does not matter to Google, but listing Allow exceptions first, as the generator does, makes the file easier for people to read.

How to test and upload your robots.txt file

Save the file with the exact name robots.txt, all lower case, as plain UTF-8 text. Upload it to the web root so it loads at https://yourdomain/robots.txt; a file in a subfolder is ignored. If you use www and non-www, or several subdomains such as shop.example.co.uk, each host needs its own file.

After uploading, open the file in a private browser window to make sure you see the new version rather than a cached one. In Google Search Console, the robots.txt report under Settings shows which version Google last fetched and any parsing problems. Bing Webmaster Tools has a similar robots.txt tester. Then check one or two important pages with the URL Inspection tool to confirm they are still allowed.

Robots.txt best practice for small business websites

Keep the file short. Every rule is something that can go wrong later, and most small websites need only a handful of lines. Block areas, not individual pages, and remember that rules are prefix matches: Disallow: /search also blocks /search-engine-optimisation/. Add a trailing slash when you mean a folder.

Never use robots.txt to hide private or sensitive pages. The file is public, and a blocked URL can still appear in results, without a description, if other pages link to it. To keep a page out of search, allow crawling and add a noindex robots meta tag, or put the page behind a login. Review your robots.txt whenever you change platform, theme or SEO plugin, and regenerate it rather than patching an old file you no longer understand.

Examples you can copy

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/

Sitemap: https://www.example.co.uk/sitemap_index.xml

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.co.uk/sitemap.xml

User-agent: *
Disallow: /

Common mistakes to avoid

Frequently asked questions