ResearchAll sizesGlobal
4.5%
of desktop websites name OpenAI's GPTBot in robots.txt, up from 2.9% in 2024
Sample: HTTP Archive crawl of millions of home pages, desktop and mobile
HTTP Archive, Web Almanac (SEO chapter), 2025 crawl
ResearchUpdated 23 Oct 2026
3.6%of desktop websites name Anthropic's ClaudeBot in robots.txt, up from 1.9% in 2024
3.6% of desktop websites name Anthropic's ClaudeBot in robots.txt, up from 1.9% in 2024 (2025 crawl, HTTP Archive, Web Almanac (SEO chapter))
ResearchChecked at source, 23 Oct 2026All sizesGlobalAI search and zero-click · AI crawlers
In plain words
What this number really says
3.6% of desktop websites name Anthropic's ClaudeBot in robots.txt, up from 1.9% in 2024 (2025 crawl).
3.6%
the latest figure (2025 crawl)
Research
type of source
Why you can trust it. HTTP Archive, Web Almanac (SEO chapter): research, research firm or analyst compilation.
Key facts
- Period 2025 crawl
- Region Global
- Company size All sizes
- Next data Yearly
Source: HTTP Archive, Web Almanac (SEO chapter), 2025 crawl
Method and use
How it was measured, and how to use it
How it was measured
- Publisher
- HTTP Archive, Web Almanac (SEO chapter)
- Source type
- Research: Research firm or analyst compilation
- What it measures
- 3.6% of desktop websites name Anthropic's ClaudeBot in robots.txt, up from 1.9% in 2024
- Data period
- 2025 crawl (ends 31 Jul 2025)
- Published by the source
- 15 Jan 2026
- In the source's words
claudebot : 3.6% (desktop), 3.4% (mobile) is up from 1.9% (desktop) and 1.6% (mobile) in 2024, nearly doubling.
- Checked at source
- 23 Oct 2026: Read on almanac.httparchive.org, Web Almanac 2025 SEO chapter (published 15 Jan 2026, updated 9 Jun 2026). 2025 crawl.
- Reuse
- Web Almanac content is open (licence not re-checked this session). Cite the Web Almanac chapter and year.
- Next expected update
- Yearly (HTTP Archive, yearly)
- Original page
- almanac.httparchive.org: Web Almanac (SEO chapter)
- Data files
- CSV · JSON (stable links)
Full change log
No changes recorded yet. First checked at the source on 23 Oct 2026.
Cite this number
APA
HTTP Archive. (2026). 3.6% of desktop websites name Anthropic's ClaudeBot in robots.txt, up from 1.9% in 2024. Via digitaltreed, https://digitaltreed.com/statistics/ai-search/sites-naming-claudebot-in-robots-txt
MLA
"3.6% of desktop websites name Anthropic's ClaudeBot in robots.txt, up from 1.9% in 2024." HTTP Archive, Web Almanac (SEO chapter), 2025 crawl. digitaltreed, https://digitaltreed.com/statistics/ai-search/sites-naming-claudebot-in-robots-txt.
HTML link
<a href="https://digitaltreed.com/statistics/ai-search/sites-naming-claudebot-in-robots-txt">3.6% of desktop websites name Anthropic's ClaudeBot in robots.txt, up from 1.9% in 2024</a> (HTTP Archive, 2025 crawl)
Embed the chart
This stat has no chart to embed yet. Link to this page instead: https://digitaltreed.com/statistics/ai-search/sites-naming-claudebot-in-robots-txt
FAQ
Questions about this number
What is the latest figure for AI crawlers (Global)?
3.6% of desktop websites name Anthropic's ClaudeBot in robots.txt, up from 1.9% in 2024, for 2025 crawl, according to HTTP Archive, Web Almanac (SEO chapter). We checked it at the source on 23 Oct 2026.
Put this number to work
We help small and mid-size firms turn data like this into useful content, answers for customers and visibility in AI search.
Every number on this page was checked at its original source. Free to cite with a link to this page.