AI Crawler Index
Live tracker of web crawler activity across the internet. Updated daily with data from Cloudflare Radar, covering billions of requests to millions of websites.
Meta-ExternalAgent
Meta
17.89%
Googlebot
22.7%
ClaudeBot
Anthropic
13.9%
Source: Cloudflare Radar — normalized request volumes across Cloudflare's global network
GPTBot
OpenAI
13.14%
Other Bots
Various
10.97%
Bingbot
Microsoft
7.92%
Amazonbot
Amazon
5.73%
facebookexternalhit
Meta
3.3%
YandexBot
Yandex
3.21%
PetalBot
Huawei
1.24%
Google's primary web crawler for indexing pages in Google Search. The most active crawler on the web.
Block via robots.txt:
User-agent: Googlebot
Meta's AI training crawler used for Llama models and Meta AI products.
Block via robots.txt:
User-agent: Meta-ExternalAgent
Anthropic's web crawler used to train Claude AI models. Respects robots.txt directives.
Block via robots.txt:
User-agent: ClaudeBot
OpenAI's web crawler used to train and improve ChatGPT and other models. Respects robots.txt directives.
Block via robots.txt:
User-agent: GPTBot
Microsoft's web crawler for indexing pages in Bing Search and powering Copilot answers.
Block via robots.txt:
User-agent: Bingbot
Amazon's crawler for Alexa AI and their machine learning services.
Block via robots.txt:
User-agent: Amazonbot
Meta's crawler that fetches page previews when URLs are shared on Facebook and Instagram.
Block via robots.txt:
User-agent: facebookexternalhit
Yandex's web crawler for indexing pages in Russia's largest search engine.
Block via robots.txt:
User-agent: YandexBot
LLM Pulse monitors how AI models mention your brand. See which crawlers visit your site and how that translates into AI visibility.