- Free, no sign-up
- Fetch live or paste rules
- Up to 10 URLs at once
Free Robots.txt Tester
See which URLs Googlebot, Bingbot, GPTBot and other crawlers may visit. Fetch a live robots.txt or paste your own rules and test up to 10 paths.
0.0 0 ratings
Full URLs work too: we test the path. Leave it empty to test the homepage.
How useful was this tool?
Be the first to rate it.
Why it matters
Why robots.txt still matters
If Google or Bing skips key pages on your site, or crawls things it should not, your robots.txt file is usually the reason.
Robots.txt is not a leftover from early SEO. It is still the first thing a well-behaved crawler reads on your site, and it tells it which parts it may visit and which to skip. When it is wrong, and plenty are, you can hide your best pages by accident or waste crawl budget on junk.
A typical case: a site stops showing up for its own brand name, and the cause turns out to be one line that blocks the whole /blog/ folder, the site's best content. One file, one wrong directive, a big drop in traffic. Testing your paths takes seconds and catches it before it costs you.
How to use it
Test your robots.txt in four steps
Pull a live robots.txt or paste your own. Either way, results appear on this page in seconds.
Pick a mode
Fetch a domain's live file, or paste rules you are still writing to test them before you deploy.
Choose a crawler
All crawlers (*), Googlebot, Googlebot-Image, Bingbot, Slurp (Yahoo), DuckDuckBot, GPTBot or facebookexternalhit.
Add URLs or paths
Up to 10, one per line. Full URLs work too: we test the path part. Leave it empty to test the homepage.
Get instant results
Green means crawlable, red means blocked, and each row shows the exact rule that decided it.
Built forSEO specialistsDevelopersSite ownersAgenciesWordPress users
Features
What the robots.txt tester does
It reads robots.txt the way Google does: the most specific rule wins, and Allow wins a tie.
Live fetch
Reads /robots.txt from the domain over HTTPS, then HTTP if needed, and shows the source URL.
Paste and test
Try new rules before you upload them, so a typo never blocks your site in production.
Matched rule
Every result names the Allow or Disallow line that applied, or says no rule matched.
Wildcards and $
Supports * for any characters and $ for end of URL, the same patterns Google and Bing support.
Whole-site warning
A clear alert when the chosen crawler is blocked from your homepage, which usually means the whole site.
Sitemaps and raw file
Lists the Sitemap lines it found and lets you view the full raw file below the results.
Quick refresher
What is in a robots.txt file?
A plain text file at the root of your domain, like https://yoursite.com/robots.txt. It is made of groups, and each group starts with a User-agent line.
User-agent: *Disallow: /private/Allow: /blog/Sitemap: https://yoursite.com/sitemap.xml
User-agent
Which crawler the group is for. * means every crawler without its own group; a name like Googlebot targets one bot.
Disallow
A folder or page the crawler should not request, such as /wp-admin/.
Allow
Permission for a specific path, even inside a broader Disallow, such as /wp-admin/admin-ajax.php.
Sitemap
The full URL of your XML sitemap. It applies to all crawlers and can sit anywhere in the file.
How rules are matched
Which rule wins?
When several lines could apply, this is the order the tester (and Google) follows.
- The crawler's own group beats
*. If Googlebot has its own group, it ignores the*group completely. A group namedgooglebotalso covers Googlebot-Image when Googlebot-Image has no group of its own. - The longest matching path wins.
Allow: /blog/beatsDisallow: /for /blog/post, because it is longer and more specific. - On a tie, Allow wins. The less restrictive rule is used.
- No matching rule means allowed. So does an empty
Disallow:line.
| Rule | Path tested | Result |
|---|---|---|
Disallow: /private | /private-notes.html | Blocked: rules match from the start of the path, so it covers anything beginning with /private |
Disallow: /private/ | /private | Allowed: the trailing slash makes it a folder rule |
Disallow: /*.pdf$ | /files/guide.pdf | Blocked: * matches any characters, $ marks the end |
Disallow: /*.pdf$ | /files/guide.pdf?v=2 | Allowed: the URL no longer ends in .pdf |
Disallow: /Blog/ | /blog/ | Allowed: paths are case-sensitive |
Common mistakes
Common issues that cause big headaches
Most people assume their robots.txt is fine until it quietly hurts their SEO. These are the mistakes we see most.
"Disallow: /" left from staging
One slash blocks the entire site. If the tester shows the whole-site warning, remove that line or change it to an empty Disallow:, which allows everything.
Wrong location
Crawlers only read robots.txt at the root: yoursite.com/robots.txt, not /assets/robots.txt or /admin/robots.txt. A copy in a subfolder is ignored.
File not in plain text
It must be a plain .txt file named exactly robots.txt (lowercase). No .html, no Word formatting, no curly quotes.
Case sensitivity
Paths are case-sensitive: Disallow: /Private/ does not block /private/. Test both spellings if your URLs mix cases. (User-agent names are not case-sensitive.)
Empty Disallow lines
An empty Disallow: does not block anything; it allows everything. If you meant to block a folder, put the path after the colon.
Stray blank lines or text
Use blank lines only between groups. Google ignores blank lines inside a group, but some older crawlers read one as the end of the group, and a typo like Dissallow is silently ignored by everyone.
Wildcard misuse
One misplaced asterisk can block a whole section. Disallow: /*? blocks every URL with a query string, including filtered category pages you may want crawled. Test a few real URLs after any wildcard change.
Using it to hide pages
Robots.txt stops crawling, not indexing. A blocked URL can still appear in Google if other pages link to it. Use noindex for that, and a login for private content.
Best practices
Robots.txt best practices
If you are creating or editing your robots.txt, this is what works.
One file per host
Each subdomain (and http vs https) needs its own robots.txt. blog.yoursite.com does not use the file on www.yoursite.com.
One directive per line
Start each directive on a new line and keep groups tidy, so the file is easy to read and audit.
Do not hide sensitive data with it
Robots.txt is public. Anyone can read it, and listing a secret folder points people to it. Protect private content with a login.
Use Allow carefully
Only when you need to open a path inside a broader Disallow. Extra Allow lines make the file harder to reason about.
Block folders, not files
Avoid listing single files unless you really must. Folder rules are shorter and survive new pages being added.
Use noindex to remove pages
Robots.txt alone does not remove indexed pages. Add a noindex tag, and leave the page crawlable so Google can see it.
FAQ
Frequently asked questions
How do I know if a page is blocked from crawling?
Enter the domain and the page URL, choose the crawler, and click Test Robots.txt. The result says Allowed or Blocked and shows the rule that matched.
Can bots ignore robots.txt?
Yes. The file is a voluntary standard, not a lock or a legal contract. Google, Bing and other major crawlers follow it, but some scrapers and bad actors do not. Protect private content with a login.
What if my robots.txt is wrong?
Fix it in any plain text editor and upload it to your site root. In WordPress, SEO plugins such as Yoast or Rank Math let you edit it from the dashboard. Paste your new version here first to test it.
Is there still a robots.txt tester in Google Search Console?
Google retired the old tester. Search Console now has a robots.txt report that shows which file Google fetched and any parse errors, and the URL Inspection tool shows whether one page is blocked. This tool lets you test up to 10 paths against 8 crawlers at once, and test rules before you publish them.
What happens if my site has no robots.txt?
Crawlers treat that as permission to crawl everything. The tester tells you when it finds no file (a 404).
Can I redirect my robots.txt file?
Crawlers only look for it at /robots.txt on each host. Google follows a few redirects for it, but other crawlers may not, so keep the real file at the root to be safe.
Will Google respect my robots.txt settings?
Yes, Googlebot follows them. Be careful when blocking CSS and JavaScript files, though: if Google cannot load the files it needs to render the page, it may not understand the page properly, which can hurt rankings. Robots.txt also does not stop a blocked URL from being indexed if it is linked from elsewhere.
Is pasted content sent anywhere?
It is sent to our server only to run the test, then discarded. It is never fetched from or sent to any other site.
Keep going
More free SEO tools
- AI Crawler Checker: see which of 116 crawlers, including ChatGPT and Claude, your robots.txt lets in.
- SSL Checker: confirm your certificate is valid and not about to expire.
- HTTP Status Checker: see status codes and redirect chains.
- Canonical Tag Checker: make sure the right URL gets the credit.
- Noindex Tag Checker: find pages kept out of search by a noindex.
- Schema Markup Validator: check your structured data for errors.
- Meta Tag Analyzer: review titles, descriptions and social tags.
Did this tool help you?
Tap a star. It takes a second, no sign-up.
–
0.0 0 ratingsNo ratings yet.
Run your URLs through it now
One line can block your whole site. Check after every redesign, migration or plugin update. Free, no sign-up.