Free AI Crawler Checker
Also called an AI search visibility checker, AI readiness checker, GEO checker, or GPTBot checker.
Find out if ChatGPT, Claude, Perplexity, and Gemini can actually read your site — before you spend another hour on content they can’t even see.
Checks robots.txt against 116 crawlers across 8 categories — AI bots, search engines, Google bots, SEO tools, social bots, scrapers, cloud services, and other agents — plus llms.txt and technical AI-readiness signals.
You Could Be Invisible to AI and Not Know It
A surprising number of sites block GPTBot or PerplexityBot in robots.txt without ever meaning to — a security plugin default, a “block all bots” rule copied from an old tutorial, a firewall preset. Nothing breaks, no error shows up in Search Console, and traffic from AI answers simply never arrives.
Traditional SEO asks “will Google rank this page?” AI search visibility asks a different question: “can an AI assistant read this page well enough to cite it?” A site can rank perfectly on Google and still be invisible to ChatGPT, Claude, and Perplexity — because those tools use separate crawlers with separate permissions in your robots.txt.
116 Crawlers Across 8 Tiers
Every bot is independently researched from vendor documentation and checked with the same longest-match rule order search engines use.
Which AI Bots Actually Matter
Not every AI bot does the same job. GPTBot collects training data for OpenAI’s models, but OAI-SearchBot is the one that powers a live ChatGPT Search answer. Block the wrong one and you’ve solved nothing; block the right one and you’ve quietly opted out of an entire answer engine.
How the AI Readiness Score Works
No hidden formula — the full breakdown is right here, and the math is the same for every site.
Only Tier 1 AI bots count toward Bot Access. SEO tools, social bots, and monitoring services are shown in full for context, but they don’t affect whether ChatGPT or Claude can cite you — so they don’t dilute the score.
Common Ways Sites Block AI by Accident
The blanket disallow
Meant to stop one scraper, it blocks every bot — including the ones that would cite you.
User-agent: * Disallow: /
A header you never wrote
Some hosts and security plugins add an AI opt-out header to every response.
X-Robots-Tag: noai
An outdated bot name
Vendors rename crawlers. A rule for a retired token may no longer match anything.
User-agent: anthropic-ai
Blocking training, not search
Blocking GPTBot alone won’t remove you from ChatGPT Search — but blocking OAI-SearchBot will.
User-agent: OAI-SearchBot Disallow: /
A firewall that answers first
Bot protection can challenge an AI crawler before robots.txt is even read.
403 Forbidden
No discovery signals
Missing sitemap, schema, or llms.txt leaves an AI assistant guessing which pages matter.
GET /llms.txt → 404
What Is llms.txt, and Do You Need One?
llms.txt is a proposed standard — a plain-text file at your domain root that summarizes what your site is and points AI assistants to your most important pages in a clean, easy-to-parse format. The optional llms-full.txt goes further, embedding fuller page content directly.
To be clear: no major AI vendor has publicly confirmed reading llms.txt yet. It’s an emerging convention, not a ranking factor. It takes minutes to add, though, and this checker tells you whether you have one.
GEO vs SEO: What’s the Difference?
SEO (Search Engine Optimization) is about ranking in classic search results and earning clicks. GEO (Generative Engine Optimization) is about whether AI answer engines can access, understand, and cite your content when they generate a response. The two overlap — crawlability, structured data, and clear content help both — but a site can be strong in one and weak in the other. This checker covers the access layer of GEO: the first thing that has to be right before anything else matters.
How to Use This Checker
Enter Your URL
Any public site — yours, a client’s, or a competitor’s.
Review the Tiers
Expand any tier to see each bot, its operator, and the exact rule that matched.
Copy the Fix
Blocked bots come with ready-to-paste robots.txt lines.
Accuracy & Limitations
✓ What this gives you
- 116 independently researched crawler tokens across 8 tiers
- The exact matched robots.txt rule for every bot, not just a pass/fail
- A transparent, fully explained scoring formula
✕ What it can’t guarantee
- robots.txt is a voluntary standard (RFC 9309), not real access control
- A firewall or WAF can still block a bot that robots.txt allows
- Being accessible doesn’t guarantee an AI engine will cite you — content quality still matters
Frequently Asked Questions
Is this AI crawler checker free?
Yes — free, no sign-up, no limit on how many sites you check.
How do I know if ChatGPT can access my website?
Enter your URL and open Tier 1. Check OAI-SearchBot (live ChatGPT Search) and ChatGPT-User (on-demand fetches). If either shows “Blocked,” ChatGPT can’t read that content.
What’s the difference between GPTBot and OAI-SearchBot?
GPTBot gathers data for OpenAI’s model training. OAI-SearchBot powers live ChatGPT Search results. Blocking GPTBot doesn’t remove you from ChatGPT Search; blocking OAI-SearchBot does.
Should I block AI crawlers?
It depends on your goals. If you want to appear in AI answers, allow the search and retrieval bots. If you want to keep content out of model training, you can block training-only bots like GPTBot or CCBot while still allowing the search ones.
What is a GEO score?
A measure of how well AI answer engines can access and understand your site — separate from traditional SEO ranking.
Do I need an llms.txt file?
It’s optional and unconfirmed by major AI vendors, but low-effort to add. This tool flags whether you have one.
Why does the score only count Tier 1 AI bots?
Because those are the crawlers that determine AI search visibility. Other tiers are shown for a complete picture but don’t affect whether an AI assistant can cite you.
Can I check a competitor’s site?
Yes — it reads publicly available files, so it works on any public URL.
More Free SEO Tools
Find Out If You’re Actually Visible
Check your site against 116 crawlers in seconds.