Free Robots.txt Tester — Check Which URLs Search Engines Can Crawl
If Google or Bing is skipping key pages on your site — or crawling stuff it shouldn’t — your robots.txt file is usually why. Test it in seconds, no login required.
Works with a live domain or your own pasted rules — no account needed.
Why Robots.txt Still Matters
A robots.txt file isn’t a leftover from early SEO — it’s still your first line of communication with search engine bots. It tells them which parts of your site they can access and which parts to skip. If it’s misconfigured, and plenty are, you could be accidentally hiding important pages or wasting crawl budget on junk.
How to Use This Tool
Pull a live robots.txt or paste your own — either way, results appear on this page in seconds.
Pick a Mode
Fetch a domain’s live robots.txt, or paste your own rules to test before deploying.
Choose a Crawler
Test against all bots (*), or a specific one like Googlebot or Bingbot.
Add Your URLs
Paste up to 10 URLs or paths, one per line.
Get Instant Results
Green means crawlable, red means blocked — plus the exact rule that decided it.
Quick Refresher: What’s in a robots.txt File?
A robots.txt file is a plain text file that sits at the root of your domain — like https://yoursite.com/robots.txt. It uses simple directives to tell bots what they can and can’t crawl.
Disallow: /private/
Allow: /blog/
Sitemap: https://yoursite.com/sitemap.xml
User-agentDefines which bot the rule applies to. Use * for all bots, or name a specific one like Googlebot.
DisallowBlocks crawlers from accessing certain folders or pages.
AllowGives bots permission to crawl a specific path, even inside a broader Disallow.
Common Issues That Cause Big Headaches
Most people assume their robots.txt is fine — until it quietly hurts their SEO. Here are the most frequent mistakes we see.
Wrong location
The file must sit at the root — yoursite.com/robots.txt, not in /assets/ or /admin/, or bots won’t find it.
File not in plain text
It must be a .txt file — no .html, no rich formatting.
Case sensitivity
URLs are case-sensitive, and bots notice. /Blog/ and /blog/ are not the same path.
Empty Disallow lines
An empty Disallow: doesn’t block anything — it allows everything.
Random blank lines or spaces
Blank lines should only separate sections by user-agent. Misplaced spaces can make bots ignore your rules entirely.
Wildcard misuse
One misplaced asterisk (*) can unintentionally block an entire section of your site.
Best Practices
If you’re creating or editing your robots.txt file, here’s what actually works.
One file per domain
Subdomains need their own robots.txt.
Start each directive on a new line
Keep the structure clean and easy to audit.
Don’t try to hide sensitive data this way
Robots.txt is publicly visible — if something’s truly private, protect it with authentication instead.
Use Allow carefully
Only when you actually need to override a broader Disallow.
Block folders, not individual files
Avoid listing single files unless it’s absolutely necessary.
Use meta noindex to keep pages out of search results
Robots.txt alone doesn’t remove indexed pages — pair it with a noindex tag when needed.
Frequently Asked Questions
Can bots ignore robots.txt?
Yes. The file is a guideline, not a legal contract. Legitimate bots like Googlebot will obey it, but some scrapers or bad actors won’t.
What if my robots.txt is wrong?
Fix it in any plain text editor and re-upload it to your root directory. If you’re using WordPress, plugins like Rank Math or Yoast let you edit it from the dashboard.
Is there still a robots.txt tester in Google Search Console?
Not in the latest version, but you can still test URLs manually — or just use our tool. It’s quicker and shows live results.
How do I know if a page is blocked from crawling?
Just enter the URL into our tester. We’ll show you exactly what the bots see.
Can I redirect my robots.txt file?
Technically no. Bots expect to find it at /robots.txt on your domain. Redirecting it elsewhere might cause it to be ignored.
Will Google respect my robots.txt settings?
Usually, yes. But if Google thinks a blocked resource is essential — like a JS file required for rendering — it might still try to access it. Google reserves the right to ignore rules that impact usability or violate its guidelines.
More Free SEO Tools
Run Your URLs Through It Now
A single line can block your entire site from being crawled. Check after every redesign, migration, or plugin update — before it becomes a disaster.