- Free, no sign-up
- 120 crawlers checked
- Copy-paste fix code
Free AI Crawler Checker
Can ChatGPT, Claude, Perplexity and Gemini read your site? Test your robots.txt against 120 crawlers, see search, user-fetch and training bots apart, and get an AI readiness score.
5.0 2 ratings
340+ uses · 56 this week
Checks robots.txt against 120 crawlers in 8 categories (AI bots, search engines, Google bots, SEO tools, social bots, scrapers, cloud services and other agents). The score counts the bots that let AI assistants find and cite you; blocking training bots is your choice and is not scored.
How useful was this tool?
Used 340+ times here · free forever
The hidden problem
You could be invisible to AI and not know it
This tool is also known as an AI search visibility checker, AI readiness checker, GEO checker or GPTBot checker. It answers one question: can ChatGPT, Claude, Perplexity and Gemini actually read your site?
Plenty of sites block GPTBot or PerplexityBot in robots.txt without ever meaning to. A security plugin turns it on by default, a "block all bots" rule gets copied from an old tutorial, or a firewall preset does it for you. Nothing breaks. No error shows up in Search Console. Visitors from AI answers simply never arrive.
Classic SEO asks "will Google rank this page?" AI search visibility asks something else: "can an AI assistant read this page well enough to cite it?" You can rank well on Google and still be invisible to ChatGPT, Claude and Perplexity, because those tools use their own crawlers, with their own permissions in your robots.txt.
How to use it
Check AI access in three steps
Enter a URL
Your homepage works best. Any public site works, so you can check a client's or a competitor's site too.
Read your score and open the tiers
You get a score out of 100 with two parts: access for the search and user-fetch bots, and technical readiness. Three lanes show search bots, user fetches and training bots; training is your choice and is not scored. Expand any tier to see each bot, who runs it, and the exact robots.txt rule that decided its verdict. You can also search all 120 bots by name.
Copy the fix
Blocked AI bots come with ready-to-paste robots.txt lines. Add them to your file, then run the check again to confirm.
Built forSite ownersSEO and GEO specialistsContent marketersAgenciesDevelopers
What it checks
120 crawlers in 8 tiers
Every bot's user-agent token comes from its vendor's documentation. Your robots.txt is checked the way Google reads it: the most specific user-agent group applies, then the longest matching rule wins, and Allow beats Disallow on a tie.
| Tier | Bots | Examples |
|---|---|---|
| 1 · AI bots | 43 | GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Meta-WebIndexer, Amzn-SearchBot, CCBot, MistralAI-Index |
| 2 · Search engines | 10 | Bingbot, DuckDuckBot, YandexBot, Baiduspider, Applebot, Naver's Yeti |
| 3 · Google bots | 15 | Googlebot, Googlebot-Image, Googlebot-News, AdsBot-Google, Storebot-Google |
| 4 · SEO tools | 20 | AhrefsBot, SemrushBot, MJ12bot, DotBot, Screaming Frog, Chrome-Lighthouse |
| 5 · Social bots | 12 | facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, WhatsApp, Discordbot |
| 6 · Scrapers and data crawlers | 10 | ia_archiver, archive.org_bot, 80legs, Omgilibot, img2dataset |
| 7 · Cloud and monitoring | 5 | UptimeRobot, Pingdom, StatusCake, Google Cloud uptime checks |
| 8 · Other agents | 5 | HeadlessChrome, Google-Agent, FirecrawlAgent, Devin |
llms.txt and llms-full.txt
Whether your site serves these optional files. Shown for information; they do not change the score.
Technical signals
HTTPS, an XML sitemap and Schema.org structured data on the page. A noai X-Robots-Tag header is shown too, but not scored.
Exact reason per bot
Not just pass or fail: each crawler shows which robots.txt group and rule decided its verdict.
Know your bots
Which AI bots actually matter
AI bots do different jobs. Block the wrong one and you have solved nothing. Block the right one and you have quietly opted out of a whole answer engine.
| Bot | Run by | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content to train OpenAI's models |
| OAI-SearchBot | OpenAI | Finds pages to show and cite in ChatGPT Search |
| ChatGPT-User | OpenAI | Fetches a page on demand when a ChatGPT user asks about it |
| ClaudeBot | Anthropic | Collects content that may be used to train Anthropic's models |
| Claude-SearchBot | Anthropic | Indexes pages to improve Claude's search answers |
| Claude-User | Anthropic | Fetches a page on demand for a Claude user |
| PerplexityBot | Perplexity | Search retrieval; Perplexity cites sources by name in its answers |
| Google-Extended | Not a crawler but a token: it opts your content out of Gemini training and grounding. It does not affect Google Search or AI Overviews | |
| Applebot-Extended | Apple | Also a token: opts your content out of Apple's foundation model training |
| Bingbot | Microsoft | Builds the Bing index, which also powers Microsoft Copilot answers (it sits in the search engines tier) |
The score
How the AI readiness score works
No hidden formula. The math is the same for every site: Score = Search and answer access (70) + Tech readiness (30).
| Signal | Points | What it checks |
|---|---|---|
| Search and answer access | 70 | The share of the 17 search and user-fetch bots your robots.txt allows: the AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot and others), the bots that fetch a page for a user (ChatGPT-User, Claude-User and others), plus Googlebot, Bingbot and Applebot, which feed AI Overviews, Copilot and Apple's answers. |
| HTTPS | 10 | The URL you entered is served over HTTPS |
| Sitemap | 10 | A sitemap.xml at the root, or a Sitemap line in robots.txt |
| Schema.org | 10 | JSON-LD or microdata found on the page you entered |
Training bots such as GPTBot, ClaudeBot, Google-Extended and CCBot are not scored. Blocking them keeps your content out of model training and does not remove you from AI answers, so it is a valid choice; the result shows how many you block. llms.txt, llms-full.txt and a noai header are shown for information only: Google says llms.txt will neither harm nor help. SEO tools, social bots and monitoring services are shown in full for context. 85 or more reads as AI-ready, 60 to 84 as partly blocked, and below 60 most AI assistants cannot use your content.
Common mistakes
How sites block AI by accident
Most blocks are not a choice. They come from a plugin, a template or an old rule nobody remembers.
The blanket disallow
Meant to stop one scraper, User-agent: * with Disallow: / blocks every bot that has no group of its own, including the ones that would cite you. Fix: give the bots you want their own group with Allow: /, or name only the bot you want to stop.
A header you never wrote
Some hosts, CDNs and security plugins add X-Robots-Tag: noai to every response. You will not see it in your page source. This checker reads the header and shows it. It only asks not to be used for training, so it does not change the score.
An outdated bot name
Vendors rename crawlers. A rule for an old token like anthropic-ai or Claude-Web may no longer match the bot that visits today. Write rules for the current names, such as ClaudeBot and Claude-SearchBot.
Blocking search to stop training
Blocking GPTBot alone will not remove you from ChatGPT Search, but User-agent: OAI-SearchBot with Disallow: / will. You can block the training crawlers and still allow the search ones, so you stay citable.
A firewall that answers first
A WAF or "bot fight" mode can answer an AI crawler with 403 Forbidden before robots.txt is even read. If an allowed bot never visits, check your firewall and CDN logs.
No discovery signals
No sitemap and no structured data leave an AI assistant guessing which pages matter and what they are about. Each of these adds points to your score.
AI files
What is llms.txt, and do you need one?
llms.txt is a proposed standard: a plain-text (Markdown) file at your domain root that says what your site is and points AI assistants to your most important pages in a clean, easy-to-read format. The optional llms-full.txt goes further and puts fuller page content straight into the file.
To be clear, no major AI company has publicly confirmed that it reads llms.txt yet. It is an emerging convention, not a ranking factor, and Google says it will neither harm nor help in Search. It takes only minutes to add and can help developer and docs sites, so this checker tells you whether you have each file, but it does not add or take away points.
GEO vs SEO
GEO vs SEO: what is the difference?
SEO (search engine optimisation) is about ranking in classic search results and earning clicks. GEO (generative engine optimisation) is about whether AI answer engines can access, understand and cite your content when they write a response.
The two overlap. Crawlability, structured data and clear content help both. Still, a site can be strong in one and weak in the other. This checker covers the access layer of GEO: the first thing that has to be right before anything else matters.
Why it matters
Why AI crawler access matters
People now ask ChatGPT, Claude, Perplexity and Gemini the questions they used to type into a search box.
Citations in AI answers
An AI assistant can only quote and link to pages its crawler may read. A blocked site is simply left out.
Ranking is not enough
You can rank well on Google and still be invisible to AI search. GEO starts with access.
Control on your terms
Seeing every bot by name lets you decide on purpose: allow the ones that send visitors, block the ones you do not want.
Honest limits
Accuracy and limitations
What this gives you
- 120 crawler tokens across 8 tiers, each taken from vendor documentation where the vendor publishes one.
- The exact matched robots.txt rule for every bot, not just a pass or fail.
- A transparent scoring formula, fully explained above.
- Ready-to-paste robots.txt lines for blocked AI bots.
What it cannot guarantee
- Robots.txt is a voluntary standard (RFC 9309), not real access control. Well-behaved bots follow it; others may not.
- A firewall or WAF can still block a bot that robots.txt allows. This tool reads your rules; it does not see your firewall.
- Rules are tested for your homepage path (/). A rule that blocks only one folder will not change the score.
- If robots.txt returns a server error (5xx), crawlers treat that as "disallow all", and so does this checker.
- Being accessible does not mean an AI engine will cite you. Content quality still matters.
FAQ
Frequently asked questions
Is this AI crawler checker free?
Yes. It is free with no sign-up. A fair-use limit of 10 checks a minute and 100 an hour keeps it fast for everyone, which is plenty for checking your own sites and your competitors'.
How do I know if ChatGPT can access my website?
Enter your URL and open tier 1. Look at OAI-SearchBot (ChatGPT Search) and ChatGPT-User (on-demand fetches when a user shares a link). If both are allowed, ChatGPT can read your pages. If either shows "Blocked", that route is closed. GPTBot only affects model training.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content to train OpenAI's models. OAI-SearchBot finds pages to show and cite in ChatGPT Search. Blocking GPTBot does not remove you from ChatGPT Search; blocking OAI-SearchBot does.
Should I block AI crawlers?
It depends on your goal. If you want to be cited in AI answers, allow the search and user-fetch crawlers. If you want to keep your content out of model training, block training tokens such as GPTBot, Google-Extended and CCBot. Many sites do both.
What is a GEO score?
A measure of how well AI answer engines can access and understand your site, separate from traditional SEO ranking. Our AI readiness score is one such measure, built from search and answer bot access and technical signals.
Do I need an llms.txt file?
It is optional. llms.txt is a proposed standard, and major AI companies have not confirmed they use it. It takes a few minutes to add and does no harm, and this tool shows whether you have one, but it does not change your score.
Why does the score not count training bots?
Because blocking them is a choice, not a mistake. Training bots decide whether your content trains AI models; search and user-fetch bots decide whether AI assistants can find and cite you. The score measures only the second, and the result shows the first so you can decide on purpose.
Which pages does it check?
Robots.txt rules are tested for your homepage path (/). The schema and noai checks read the exact URL you enter.
If a bot shows as allowed, will it crawl my site?
Not necessarily. An allowed bot may never visit, and a firewall can still block it. A blocked, well-behaved bot will stay away.
Can I check a competitor's site?
Yes. The tool only reads publicly available files, so it works on any public URL.
Do you store the URLs I check?
No. A result is kept for a few minutes so a repeat check is instant, then it is gone.
Keep going
More free SEO tools
- Robots.txt Tester: test any URL against any user-agent and see the rule that decides it.
- Schema Markup Validator: check the structured data AI and search engines read.
- Meta Tag Analyzer: review titles, descriptions and social tags.
- Noindex Tag Checker: find pages hidden from search by a noindex.
- Canonical Tag Checker: make sure the right URL gets the credit.
- Website Speed Test: measure load time and Core Web Vitals.
Did this tool help you?
Tap a star. It takes a second, no sign-up.
–
5.0 2 ratingsThe average shows from 5 ratings.
Find out if you're actually visible
Free, no sign-up. Check 120 crawlers and get a copy-paste fix in seconds.