By Aijaz Alam
How many websites block AI crawlers or use llms.txt?
4.5% of desktop websites named OpenAI's GPTBot in their robots.txt file in the HTTP Archive's 2025 crawl, up from 2.9% in 2024, and 3.6% named Anthropic's ClaudeBot. Naming a bot sets a rule for it, which can be a block or an allow. Only 2.13% of desktop sites had a valid llms.txt file.
- 4.5%Desktop websites naming GPTBot in robots.txt, up from 2.9% in 2024HTTP Archive Web Almanac, 2025 crawl
- 3.6%Desktop websites naming ClaudeBot in robots.txt, up from 1.9%HTTP Archive Web Almanac, 2025 crawl
- 2.13%Desktop websites with a valid llms.txt file (2.10% of mobile)HTTP Archive Web Almanac, 2025 crawl
- 39.6%llms.txt files made by the All in One SEO pluginHTTP Archive Web Almanac, 2025 crawl
- 97%llms.txt files that got no requests at all in May 2026Ahrefs, 38,000 domains with a valid file (vendor)
The numbers
The full breakdown
Websites naming AI-related crawlers in robots.txt (2025 crawl)
| Crawler (operator) | Desktop sites | Mobile sites |
|---|---|---|
| GPTBot (OpenAI) | 4.5% | 4.2% |
| PetalBot (Huawei) | 4.0% | 4.4% |
| ClaudeBot (Anthropic) | 3.6% | 3.4% |
| CCBot (Common Crawl) | 3.5% | 3.2% |
| Google-Extended (Google) | 3.4% | 3.0% |
| Amazonbot (Amazon) | 3.3% | 3.0% |
| FacebookBot (Meta) | 2.9% | 2.5% |
| PerplexityBot (Perplexity) | 2.8% | 2.7% |
| ChatGPT-User (OpenAI) | 2.8% | 2.5% |
| Meta-ExternalAgent (Meta) | 2.8% | 2.5% |
HTTP Archive Web Almanac 2025, SEO chapter, figure 5.5. Counts sites whose robots.txt names the user agent, whether the rule blocks or allows it. The Almanac lists PetalBot among the AI-related crawlers.
AI crawlers named in robots.txt, 2024 and 2025
| Crawler | Desktop 2024 | Desktop 2025 | Mobile 2024 | Mobile 2025 |
|---|---|---|---|---|
| GPTBot | 2.9% | 4.5% | 2.7% | 4.2% |
| ClaudeBot | 1.9% | 3.6% | 1.6% | 3.4% |
| CCBot | 2.7% | 3.5% | 2.4% | 3.2% |
| ChatGPT-User | 2.0% | 2.8% | 1.7% | 2.5% |
HTTP Archive Web Almanac 2025 and 2024 crawls. The 2024 data also listed anthropic-ai (2.0% desktop, 1.7% mobile), a broader Anthropic name; 2025 counts the more specific bots.
llms.txt: who publishes it, and who reads it
| Measure | Value | Source |
|---|---|---|
| Desktop websites with a valid llms.txt | 2.13% | HTTP Archive, 2025 crawl |
| Mobile websites with a valid llms.txt | 2.10% | HTTP Archive, 2025 crawl |
| Valid llms.txt files found across mobile sites | 324,184 | HTTP Archive, 2025 crawl |
| llms.txt files made by All in One SEO | 39.6% | HTTP Archive, 2025 crawl |
| llms.txt files made by Yoast SEO | 3.6% | HTTP Archive, 2025 crawl |
| Domains using Ahrefs Web Analytics that publish llms.txt | 28% | Ahrefs, 137K domains (upper bound) |
| Of those files, the share that got no requests in May 2026 | 97% | Ahrefs, about 38,000 domains |
The plugin shares come from the default comments the plugins leave in the files. Ahrefs' sample is its own analytics customers, who are more SEO-aware than the web as a whole, so Ahrefs calls its 28% an upper bound.
What websites return when asked for robots.txt (2025 crawl)
| Response | Desktop sites | Mobile sites |
|---|---|---|
| 200 (file served) | 84.9% | 84.9% |
| 404 (no file) | 13.3% | 13.2% |
| Timeout | 1.0% | 1.1% |
| 403 (forbidden) | 0.5% | 0.5% |
| 500 (server error) | 0.1% | 0.1% |
HTTP Archive Web Almanac 2025, SEO chapter, figure 5.1. A site with no robots.txt sets no rules for any crawler.
Why sources differ
Same question, different answers
| Number | When | Who measured | Why it differs |
|---|---|---|---|
| 4.5% of desktop sites | 2025 crawl | HTTP Archive | Sites whose robots.txt names GPTBot, with any rule; millions of home pages. |
| 2.13% of desktop sites | 2025 crawl | HTTP Archive | Sites with a valid llms.txt file, across the whole crawl. |
| 28% of domains | May 2026 | Ahrefs | Domains using Ahrefs Web Analytics, an SEO-aware group; Ahrefs calls it an upper bound. |
| 97% unread | May 2026 | Ahrefs | Share of valid llms.txt files that received no requests in the month. |
The two llms.txt figures are far apart because they measure different groups of sites. The HTTP Archive crawls millions of home pages across the web and finds a valid llms.txt on about 2% of them. Ahrefs looked at 137,000 domains that use its own analytics product, and found the file on 28%. Ahrefs says its customers are more technical and SEO-aware than the web at large, so it treats 28% as an upper bound. Neither is wrong; they describe different populations.
For robots.txt, there is only one large public count, the Web Almanac. It counts sites whose robots.txt names a crawler such as GPTBot. A rule can block the bot or allow it, so the figure is not a count of blocked sites.
Publishing a file is also not the same as it being used. Ahrefs found that 97% of the llms.txt files it could see got no requests in May 2026, and Google has said it does not use llms.txt.
What it means
How to use these numbers
Most websites have not changed anything for AI crawlers. In the 2025 crawl, about 95% of desktop sites did not name GPTBot in robots.txt at all, so OpenAI's crawler follows whatever general rules those sites set for every bot.
The small minority that do act is growing fast. The share naming GPTBot rose by more than half in a year, and the share naming ClaudeBot nearly doubled.
llms.txt is still an experiment. About 2% of sites have one, and many of those files were created by default by SEO plugins rather than by a deliberate decision. Ahrefs found that almost none were read.
For site owners, decide on robots.txt rules for each AI crawler you care about, and check your server logs before spending time on llms.txt.
Method and sources
How it was measured
Method
- HTTP Archive Web Almanac
- 2025 SEO chapter (published 15 Jan 2026, updated 9 Jun 2026), based on the HTTP Archive crawl of millions of home pages, desktop and mobile; 2024 figures from the same chapter
- Ahrefs
- llms.txt study published 15 Jun 2026 (updated 27 Aug 2026): server logs and live traffic of 137K domains using Ahrefs Web Analytics and Bot Analytics, requests in May 2026; vendor study
- Checked at source
- 23 Oct 2026 (repo stats); 31 Oct 2026, the Web Almanac 2025 SEO chapter and the Ahrefs study
- Last changed
- 31 Oct 2026
- Data files
- CSV · JSON (stable links)
Sources
- Web Almanac (SEO chapter), HTTP Archive Research
- AI Overviews click-through studies, Ahrefs Vendor data
Cite this page
APA
MLA
HTML link
FAQ
More questions people ask
Which AI crawler do websites name most often in robots.txt?
OpenAI's GPTBot, named by 4.5% of desktop sites and 4.2% of mobile sites in the 2025 crawl. PetalBot was most named on mobile, at 4.4%.
Does Google use llms.txt?
No. The Web Almanac notes that Google has often stated it does not use llms.txt and that no Google service currently does.
Who reads llms.txt files?
Mostly bots that are not AI tools. In Ahrefs' May 2026 data, 96% of requests to llms.txt came from bots, and only 19.5% came from named AI tools.
How many websites have no robots.txt at all?
About 13%. In the 2025 crawl, 13.3% of desktop sites and 13.2% of mobile sites returned a 404 when asked for robots.txt.
Every number on this page was checked at its original source (HTTP Archive, Ahrefs). Free to cite with a link to this page.