AI Crawler Index: Which Bots Crawl the Web Most?

An AI crawler is an automated bot that downloads web pages to power AI products. Some collect training data for large language models (GPTBot, ClaudeBot, Meta-ExternalAgent), while others index content for search and AI assistants (Googlebot, Bingbot, Applebot).

Googlebot is consistently the most active crawler on the web. Among AI training bots, GPTBot (OpenAI), ClaudeBot (Anthropic) and Meta-ExternalAgent (Meta) generate the largest crawl volumes, and their share keeps growing.

It depends on your goals: blocking AI training bots via robots.txt keeps your content out of future model training, but it can also reduce how accurately AI assistants describe your brand. Most brands that care about AI visibility allow them.

The index is built from Cloudflare Radar bot traffic data, normalized across Cloudflare's global network, and refreshed daily by LLM Pulse.