- Free, no sign-up
- 116 crawlers checked
- Copy-paste fix code
Free AI Crawler Checker
Can ChatGPT, Claude, Perplexity and Gemini read your site? Test your robots.txt against 116 crawlers, plus llms.txt and AI-readiness signals.
5.0 2 ratings
Checks robots.txt against 116 crawlers in 8 categories (AI bots, search engines, Google bots, SEO tools, social bots, scrapers, cloud services and other agents), plus llms.txt and technical AI-readiness signals.
How useful was this tool?
The hidden problem
You could be invisible to AI and not know it
This tool is also known as an AI search visibility checker, AI readiness checker, GEO checker or GPTBot checker. It answers one question: can ChatGPT, Claude, Perplexity and Gemini actually read your site?
Plenty of sites block GPTBot or PerplexityBot in robots.txt without ever meaning to. A security plugin turns it on by default, a "block all bots" rule gets copied from an old tutorial, or a firewall preset does it for you. Nothing breaks. No error shows up in Search Console. Visitors from AI answers simply never arrive.
Classic SEO asks "will Google rank this page?" AI search visibility asks something else: "can an AI assistant read this page well enough to cite it?" You can rank well on Google and still be invisible to ChatGPT, Claude and Perplexity, because those tools use their own crawlers, with their own permissions in your robots.txt.
How to use it
Check AI access in three steps
Enter a URL
Your homepage works best. Any public site works, so you can check a client's or a competitor's site too.
Read your score and open the tiers
You get a score out of 100 with three parts: AI bot access, AI files and technical readiness. Expand any tier to see each bot, who runs it, and the exact robots.txt rule that decided its verdict. You can also search all 116 bots by name.
Copy the fix
Blocked AI bots come with ready-to-paste robots.txt lines. Add them to your file, then run the check again to confirm.
Built forSite ownersSEO and GEO specialistsContent marketersAgenciesDevelopers
What it checks
116 crawlers in 8 tiers
Every bot's user-agent token comes from its vendor's documentation. Your robots.txt is checked the way Google reads it: the most specific user-agent group applies, then the longest matching rule wins, and Allow beats Disallow on a tie.
| Tier | Bots | Examples |
|---|---|---|
| 1 · AI bots | 39 | GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Meta-ExternalAgent, Amazonbot, CCBot, DeepSeekBot |
| 2 · Search engines | 10 | Bingbot, DuckDuckBot, YandexBot, Baiduspider, Applebot, Naver's Yeti |
| 3 · Google bots | 15 | Googlebot, Googlebot-Image, Googlebot-News, AdsBot-Google, Storebot-Google |
| 4 · SEO tools | 20 | AhrefsBot, SemrushBot, MJ12bot, DotBot, Screaming Frog, Chrome-Lighthouse |
| 5 · Social bots | 12 | facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, WhatsApp, Discordbot |
| 6 · Scrapers and data crawlers | 10 | ia_archiver, archive.org_bot, 80legs, Omgilibot, img2dataset |
| 7 · Cloud and monitoring | 5 | UptimeRobot, Pingdom, StatusCake, Google Cloud uptime checks |
| 8 · Other agents | 5 | HeadlessChrome, Google-Agent, FirecrawlAgent, Devin |
llms.txt and llms-full.txt
Whether your site serves these files, the proposed way to point AI tools to your best content.
Technical signals
HTTPS, an XML sitemap, Schema.org structured data on the page, and no noai in the X-Robots-Tag header.
Exact reason per bot
Not just pass or fail: each crawler shows which robots.txt group and rule decided its verdict.
Know your bots
Which AI bots actually matter
AI bots do different jobs. Block the wrong one and you have solved nothing. Block the right one and you have quietly opted out of a whole answer engine.
| Bot | Run by | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content to train OpenAI's models |
| OAI-SearchBot | OpenAI | Finds pages to show and cite in ChatGPT Search |
| ChatGPT-User | OpenAI | Fetches a page on demand when a ChatGPT user asks about it |
| ClaudeBot | Anthropic | Anthropic's main crawler |
| Claude-SearchBot | Anthropic | Search and browsing for Claude |
| Claude-User | Anthropic | Fetches a page on demand for a Claude user |
| PerplexityBot | Perplexity | Search retrieval; Perplexity cites sources by name in its answers |
| Google-Extended | Not a crawler but a token: it lets you opt out of Gemini and Vertex AI training without leaving Google Search | |
| Applebot-Extended | Apple | Also a token: opts your content out of Apple Intelligence training |
| Bingbot | Microsoft | Builds the Bing index, which also powers Microsoft Copilot answers (it sits in the search engines tier) |
The score
How the AI readiness score works
No hidden formula. The math is the same for every site: Score = Bot access (60) + AI-ready files (20) + Tech readiness (20).
| Signal | Points | What it checks |
|---|---|---|
| Bot access | 60 | The share of the 39 tier 1 AI bots your robots.txt allows. Allow half of them and you get about 30. |
| llms.txt | 12 | The file is present at your domain root |
| llms-full.txt | 8 | The fuller companion file is present |
| HTTPS | 6 | The URL you entered is served over HTTPS |
| Sitemap | 5 | A sitemap.xml at the root, or a Sitemap line in robots.txt |
| Schema.org | 6 | JSON-LD or microdata found on the page you entered |
| No "noai" header | 3 | The X-Robots-Tag header is not silently opting you out of AI |
Only tier 1 AI bots count toward bot access. SEO tools, social bots and monitoring services are shown in full for context, but they do not affect whether ChatGPT or Claude can cite you, so they do not dilute the score. 85 or more reads as AI-ready, 60 to 84 as partly blocked, and below 60 most AI assistants cannot use your content.
Common mistakes
How sites block AI by accident
Most blocks are not a choice. They come from a plugin, a template or an old rule nobody remembers.
The blanket disallow
Meant to stop one scraper, User-agent: * with Disallow: / blocks every bot that has no group of its own, including the ones that would cite you. Fix: give the bots you want their own group with Allow: /, or name only the bot you want to stop.
A header you never wrote
Some hosts, CDNs and security plugins add X-Robots-Tag: noai to every response. You will not see it in your page source. This checker reads the header and flags it.
An outdated bot name
Vendors rename crawlers. A rule for an old token like anthropic-ai or Claude-Web may no longer match the bot that visits today. Write rules for the current names, such as ClaudeBot and Claude-SearchBot.
Blocking search to stop training
Blocking GPTBot alone will not remove you from ChatGPT Search, but User-agent: OAI-SearchBot with Disallow: / will. You can block the training crawlers and still allow the search ones, so you stay citable.
A firewall that answers first
A WAF or "bot fight" mode can answer an AI crawler with 403 Forbidden before robots.txt is even read. If an allowed bot never visits, check your firewall and CDN logs.
No discovery signals
No sitemap, no structured data and GET /llms.txt → 404 leave an AI assistant guessing which pages matter. Each of these adds points to your score.
AI files
What is llms.txt, and do you need one?
llms.txt is a proposed standard: a plain-text (Markdown) file at your domain root that says what your site is and points AI assistants to your most important pages in a clean, easy-to-read format. The optional llms-full.txt goes further and puts fuller page content straight into the file.
To be clear, no major AI company has publicly confirmed that it reads llms.txt yet. It is an emerging convention, not a ranking factor. It does take only minutes to add, it does no harm, and it is worth 20 of the 100 points here (12 for llms.txt, 8 for llms-full.txt). This checker tells you whether you have each one.
GEO vs SEO
GEO vs SEO: what is the difference?
SEO (search engine optimisation) is about ranking in classic search results and earning clicks. GEO (generative engine optimisation) is about whether AI answer engines can access, understand and cite your content when they write a response.
The two overlap. Crawlability, structured data and clear content help both. Still, a site can be strong in one and weak in the other. This checker covers the access layer of GEO: the first thing that has to be right before anything else matters.
Why it matters
Why AI crawler access matters
People now ask ChatGPT, Claude, Perplexity and Gemini the questions they used to type into a search box.
Citations in AI answers
An AI assistant can only quote and link to pages its crawler may read. A blocked site is simply left out.
Ranking is not enough
You can rank well on Google and still be invisible to AI search. GEO starts with access.
Control on your terms
Seeing every bot by name lets you decide on purpose: allow the ones that send visitors, block the ones you do not want.
Honest limits
Accuracy and limitations
What this gives you
- 116 crawler tokens across 8 tiers, each taken from vendor documentation.
- The exact matched robots.txt rule for every bot, not just a pass or fail.
- A transparent scoring formula, fully explained above.
- Ready-to-paste robots.txt lines for blocked AI bots.
What it cannot guarantee
- Robots.txt is a voluntary standard (RFC 9309), not real access control. Well-behaved bots follow it; others may not.
- A firewall or WAF can still block a bot that robots.txt allows. This tool reads your rules; it does not see your firewall.
- Rules are tested for your homepage path (/). A rule that blocks only one folder will not change the score.
- If robots.txt returns a server error (5xx), crawlers treat that as "disallow all", and so does this checker.
- Being accessible does not mean an AI engine will cite you. Content quality still matters.
FAQ
Frequently asked questions
Is this AI crawler checker free?
Yes. It is free with no sign-up. A fair-use limit of 10 checks a minute and 100 an hour keeps it fast for everyone, which is plenty for checking your own sites and your competitors'.
How do I know if ChatGPT can access my website?
Enter your URL and open tier 1. Look at OAI-SearchBot (ChatGPT Search) and ChatGPT-User (on-demand fetches when a user shares a link). If both are allowed, ChatGPT can read your pages. If either shows "Blocked", that route is closed. GPTBot only affects model training.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content to train OpenAI's models. OAI-SearchBot finds pages to show and cite in ChatGPT Search. Blocking GPTBot does not remove you from ChatGPT Search; blocking OAI-SearchBot does.
Should I block AI crawlers?
It depends on your goal. If you want to be cited in AI answers, allow the search and user-fetch crawlers. If you want to keep your content out of model training, block training tokens such as GPTBot, Google-Extended and CCBot. Many sites do both.
What is a GEO score?
A measure of how well AI answer engines can access and understand your site, separate from traditional SEO ranking. Our AI readiness score is one such measure, built from bot access, AI files and technical signals.
Do I need an llms.txt file?
It is optional. llms.txt is a proposed standard, and major AI companies have not confirmed they use it. It takes a few minutes to add, so it does little harm, and this tool flags whether you have one.
Why does the score only count tier 1 AI bots?
Because those are the crawlers that decide AI search visibility. The other tiers are shown for a complete picture, but they do not affect whether an AI assistant can cite you.
Which pages does it check?
Robots.txt rules are tested for your homepage path (/). The schema and noai checks read the exact URL you enter.
If a bot shows as allowed, will it crawl my site?
Not necessarily. An allowed bot may never visit, and a firewall can still block it. A blocked, well-behaved bot will stay away.
Can I check a competitor's site?
Yes. The tool only reads publicly available files, so it works on any public URL.
Do you store the URLs I check?
No. A result is kept for a few minutes so a repeat check is instant, then it is gone.
Keep going
More free SEO tools
- Robots.txt Tester: test any URL against any user-agent and see the rule that decides it.
- Schema Markup Validator: check the structured data AI and search engines read.
- Meta Tag Analyzer: review titles, descriptions and social tags.
- Noindex Tag Checker: find pages hidden from search by a noindex.
- Canonical Tag Checker: make sure the right URL gets the credit.
- Website Speed Test: measure load time and Core Web Vitals.
Did this tool help you?
Tap a star. It takes a second, no sign-up.
–
5.0 2 ratingsThe average shows from 5 ratings.
Find out if you're actually visible
Free, no sign-up. Check 116 crawlers and get a copy-paste fix in seconds.