- Free, no sign-up
- WordPress, Shopify, Wix, plain sites
- AI bots: training vs search
Free Robots.txt Generator
Build a correct robots.txt in a minute: a safe template for your platform, the AI-bot decision made clear, your Sitemap line filled in, and a check of the file you already have. Copy, download, test.
0.0 0 ratings
1Platform
Picks the safe defaults and tells you where the file lives.
2Keep crawlers out of safe defaults
Private and thin pages. CSS, JavaScript and images are never blocked.
3AI crawlers
A request, not a lock. Names from our AI Crawler Checker.
4Sitemap
Enter your site above and we find it; or type the address.
+ Add another · No sitemap? Make one with the XML Sitemap Generator.
5Custom rules optional
6Test a path against this file
robots.txt
Upload it to the site root so it loads at /robots.txt, then check it with the Robots.txt Tester and in Search Console › Settings › robots.txt.
How useful was this tool?
Be the first to rate it.
How to use it
Build a robots.txt in three steps
Pick a starting point
A safe template for your platform, the file you already have (we load it and show what is wrong), or a blank page.
Tick what to keep crawlers out of, decide on AI bots, confirm the sitemap
Private areas, cart and account pages, archives if you want. Three plain choices for AI crawlers, with the right names. We find your sitemap and check it loads.
Test, copy, upload
Test a path against the file, then upload it to the site root so it loads at /robots.txt. Check it with the Robots.txt Tester and in Search Console › Settings › robots.txt.
Built forWordPress and WooCommerceShopify, Wix, SquarespaceStatic sitesDevelopersSEO specialists
Quick answer
What is a robots.txt generator?
A robots.txt generator writes the small text file that tells crawlers which parts of a site not to fetch. A safe one for most sites allows everything, keeps private paths like /account/ or /wp-admin/ out, allows /wp-admin/admin-ajax.php, never blocks CSS, JavaScript or images, and ends with the sitemap's address. It can also name AI crawlers: blocking GPTBot, ClaudeBot and CCBot stops training on your pages; blocking OAI-SearchBot, Claude-SearchBot and PerplexityBot removes you from AI answers. Robots.txt controls crawling, not indexing: to keep a page out of Google, use noindex and leave the page crawlable.
Your platform
Where robots.txt lives on each platform
| Platform | Has one by default? | How to change it | What to put in it |
|---|---|---|---|
| WordPress | Yes, a virtual file | Upload a real robots.txt to the root, or edit it in Yoast › Tools › File editor or Rank Math › General settings › Edit robots.txt | Block /wp-admin/, allow admin-ajax.php, never block wp-content or wp-includes, add the sitemap |
| WooCommerce | As WordPress | As WordPress | Add /cart/, /checkout/, /my-account/ and ?add-to-cart= |
| Shopify | Yes | Edit the robots.txt.liquid template (Online Store › Themes › Edit code) | Shopify's default is fine; add AI rules and the sitemap if you change it |
| Wix | Yes | SEO settings › robots.txt editor | Mostly the default; AI rules if you want them |
| Squarespace | Yes | Cannot be edited; hide pages from search in page settings | Nothing to upload |
| Blogger | Yes | Settings › Crawlers and indexing › Custom robots.txt | The generated file, with the sitemap line |
| Webflow | Yes | Site settings › SEO › robots.txt | The generated file |
| Static HTML, custom code | No | Upload the file to the web root | The generated file. |
AI crawlers
Which AI bots to block, and which to keep
Blocking is a request in a text file; the well-known crawlers honour it. The names below are the exact tokens; "Claude", "ChatGPT" and "anthropic-ai" match nothing. Our AI Crawler Checker keeps the full list of 120.
| Group | Tokens | Blocking it means |
|---|---|---|
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You stop appearing in ChatGPT, Claude and Perplexity answers. Usually keep. |
| Fetched for a user | ChatGPT-User, Claude-User, Perplexity-User | A page a user asks the assistant to open will not load. Usually keep. |
| Training | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent | Your pages are not used to train models. Does not affect Google Search. Your choice. |
Common mistakes
The robots.txt mistakes we lint for
Disallow: / left from staging
One line blocks the whole site. The lint flags it in red, and the Tester shows every path as blocked.
Blocking CSS, JavaScript and images
Google renders pages like a browser; block the theme folder and it sees a broken page. Some generators' WordPress presets do this. Ours does not.
Noindex lines in robots.txt
Google dropped them in 2019. Use a noindex meta tag, and do not block the page, or Google never sees the tag.
Wrong bot names
"anthropic-ai" and "Claude" match nothing; the crawler is ClaudeBot. A specific group also needs its own Disallow lines: it does not inherit from *.
FAQ
Frequently asked questions
Do I need a robots.txt?
Not strictly: a site without one is crawled fully. But it is where you keep private areas out, name the sitemap, and make the AI-bot decision, so every site should have a short one.
Where do I put robots.txt?
At the root of the host: https://example.com/robots.txt. A subdomain (blog.example.com) needs its own. A file in a subfolder is ignored.
Can I use robots.txt to remove a page from Google?
No. It stops crawling, not indexing; a blocked page can still show in results from its links. Use a noindex meta tag and leave the page crawlable so Google can read it.
Should I block /wp-admin/?
Yes, with one exception: allow /wp-admin/admin-ajax.php, because themes and plugins call it from the front end. Never block /wp-content/ or /wp-includes/: Google needs the CSS and JavaScript there.
How do I block AI crawlers, and should I?
Add a group for the training bots (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, meta-externalagent) with Disallow: /. Keep the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want AI answers to cite you. Blocking training bots does not affect Google Search.
Does Google respect Crawl-delay?
No. Googlebot ignores it; set the crawl rate in Search Console if you must. Bing and Yandex read it.
Which rule wins, Allow or Disallow?
The longest matching rule. Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that file. When two rules are the same length, Allow wins.
Can I use wildcards?
Yes: * matches any run of characters, $ marks the end of the URL. Disallow: /*.pdf$ blocks every PDF. Google, Bing and most AI crawlers understand both.
Is robots.txt case sensitive?
Paths are: /Account/ and /account/ are different. User-agent names are not.
Is "Disallow:" with nothing after it the same as allowing everything?
Yes. An empty Disallow allows everything for that group. Disallow: / blocks everything.
How do I test my robots.txt?
Type a path into the test box above, or send the file to our Robots.txt Tester for up to 10 paths and any user agent. Search Console › Settings › robots.txt shows what Google last fetched.
Is the robots.txt generator free?
Yes, with no sign-up. The file is built in your browser; we fetch your live robots.txt and sitemap only when you ask, and keep nothing.
Keep going
More free SEO tools
- Robots.txt Tester: test up to 10 paths for any user agent against the live file.
- AI Crawler Checker: 120 AI crawlers, which may read your site, and a fix.
- XML Sitemap Validator and XML Sitemap Generator: the file your Sitemap line points at.
- Noindex Tag Checker: the right way to keep a page out of Google.
Did this tool help you?
Tap a star. It takes a second, no sign-up.
–
0.0 0 ratingsNo ratings yet.
Ten lines that decide what Google sees
A safe file for your platform, the AI-bot choice made right, your sitemap announced. Build it, test it, upload it. Free.