ResearchAll sizesGlobal
3.6%
of desktop websites name Anthropic's ClaudeBot in robots.txt, up from 1.9% in 2024
HTTP Archive, Web Almanac (SEO chapter), 2025 crawl
ResearchUpdated 23 Oct 2026
4.5%of desktop websites name OpenAI's GPTBot in robots.txt, up from 2.9% in 2024
4.5% of desktop websites name OpenAI's GPTBot in robots.txt, up from 2.9% in 2024 (2025 crawl, HTTP Archive, Web Almanac (SEO chapter))
ResearchChecked at source, 23 Oct 2026All sizesGlobalAI search and zero-click · AI crawlers
Websites naming AI crawlers in robots.txt (desktop)
| Group | Desktop sites, 2025 |
|---|---|
| GPTBot | 4.5% |
| PetalBot | 4% |
| ClaudeBot | 3.6% |
| CCBot | 3.5% |
| Google-Extended | 3.4% |
| Amazonbot | 3.3% |
| PerplexityBot | 2.8% |
| ChatGPT-User | 2.8% |
In plain words
What this number really says
4.5% of desktop websites name OpenAI's GPTBot in robots.txt, up from 2.9% in 2024 (2025 crawl).
4.5%
the latest figure (2025 crawl)
Research
type of source
Why you can trust it. HTTP Archive, Web Almanac (SEO chapter): research, research firm or analyst compilation. HTTP Archive crawl of millions of home pages, desktop and mobile
Key facts
- Period 2025 crawl
- Region Global
- Company size All sizes
- Next data Yearly
Source: HTTP Archive, Web Almanac (SEO chapter), 2025 crawl
Method and use
How it was measured, and how to use it
How it was measured
- Publisher
- HTTP Archive, Web Almanac (SEO chapter)
- Source type
- Research: Research firm or analyst compilation
- What it measures
- 4.5% of desktop websites name OpenAI's GPTBot in robots.txt, up from 2.9% in 2024
- Data period
- 2025 crawl (ends 31 Jul 2025)
- Published by the source
- 15 Jan 2026
- Who is counted
- HTTP Archive crawl of millions of home pages, desktop and mobile
- In the source's words
gptbot : 4.5% (desktop), 4.2% (mobile) is up from 2.9% (desktop) and 2.7% (mobile) in 2024, representing a ~55% increase.
- Checked at source
- 23 Oct 2026: Read on almanac.httparchive.org, Web Almanac 2025 SEO chapter (published 15 Jan 2026, updated 9 Jun 2026). 2025 crawl.
- Reuse
- Web Almanac content is open (licence not re-checked this session). Cite the Web Almanac chapter and year.
- Next expected update
- Yearly (HTTP Archive, yearly)
- Original page
- almanac.httparchive.org: Web Almanac (SEO chapter)
- Data files
- CSV · JSON (stable links)
Full change log
No changes recorded yet. First checked at the source on 23 Oct 2026.
Cite this number
APA
HTTP Archive. (2026). 4.5% of desktop websites name OpenAI's GPTBot in robots.txt, up from 2.9% in 2024. Via digitaltreed, https://digitaltreed.com/statistics/ai-search/sites-naming-gptbot-in-robots-txt
MLA
"4.5% of desktop websites name OpenAI's GPTBot in robots.txt, up from 2.9% in 2024." HTTP Archive, Web Almanac (SEO chapter), 2025 crawl. digitaltreed, https://digitaltreed.com/statistics/ai-search/sites-naming-gptbot-in-robots-txt.
HTML link
<a href="https://digitaltreed.com/statistics/ai-search/sites-naming-gptbot-in-robots-txt">4.5% of desktop websites name OpenAI's GPTBot in robots.txt, up from 2.9% in 2024</a> (HTTP Archive, 2025 crawl)
Embed the chart
Keep the source link under the chart. The chart updates when the number does.
FAQ
Questions about this number
What is the latest figure for AI crawlers (Global)?
4.5% of desktop websites name OpenAI's GPTBot in robots.txt, up from 2.9% in 2024, for 2025 crawl, according to HTTP Archive, Web Almanac (SEO chapter). We checked it at the source on 23 Oct 2026.
Put this number to work
We help small and mid-size firms turn data like this into useful content, answers for customers and visibility in AI search.
Every number on this page was checked at its original source. Free to cite with a link to this page.