What is robots.txt and do AI bots follow it?

Robots.txt is a public file that asks crawlers which parts of your site they may visit. Most AI companies say their bots follow it, so it is the simplest way to decide which AI bots can read your pages.

Should I block AI crawlers from my site?

Usually not. Blocking AI search crawlers such as OAI-SearchBot or PerplexityBot removes your pages from the answers that cite sources. Many sites block training crawlers only and keep search crawlers and assistants allowed.

What happens if I block GPTBot or Google-Extended?

Blocking a training crawler such as GPTBot or Google-Extended stops new pages from entering future training data. It does not remove what a model already learned, and it does not stop AI search from citing you.

Does robots.txt stop every AI bot?

No. Robots.txt is a request, not a lock. Well-known AI companies respect it, but a crawler can ignore it. To see which bots really visit, look at your server or CDN logs.

Is this robots.txt generator private?

Yes. The generator runs in your browser and nothing you enter is sent to us. Copy the result or download it as robots.txt.

How often should I update my robots.txt?

Review it whenever you add pages you want to keep private and every few months, because AI companies launch new crawlers. This generator lists the AI bots we track.

Free AI search tool

AI Robots.txt Generator

Decide which of 36 AI crawlers, search bots and assistants can read your site, and get a ready-to-upload robots.txt in seconds.

Runs in your browser No sign-up Training, search and assistant bots

Before you block

Which AI bots should you block?

AI bots fall into three groups. Blocking each group has a different cost, so decide group by group instead of blocking everything.

AI search crawlers

Index pages so AI search engines and answers can cite and link to them.

If you block them: AI search stops citing and linking to your pages.

OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot

User-triggered AI assistants

Fetch a page when a person asks an AI assistant to open or read it.

If you block them: assistants cannot open your pages for the people asking about you.

ChatGPT-User, Perplexity-User, Claude-User, Meta-ExternalFetcher

AI training crawlers

Collect pages to train AI models and build their datasets.

If you block them: future models learn less about you, while AI search can still cite you.

GPTBot, ClaudeBot, Claude-Web, anthropic-ai

AI robots.txt questions

How AI crawlers read robots.txt and what blocking them changes.

: Robots.txt is a public file that asks crawlers which parts of your site they may visit. Most AI companies say their bots follow it, so it is the simplest way to decide which AI bots can read your pages.
: Usually not. Blocking AI search crawlers such as OAI-SearchBot or PerplexityBot removes your pages from the answers that cite sources. Many sites block training crawlers only and keep search crawlers and assistants allowed.
: Blocking a training crawler such as GPTBot or Google-Extended stops new pages from entering future training data. It does not remove what a model already learned, and it does not stop AI search from citing you.
: No. Robots.txt is a request, not a lock. Well-known AI companies respect it, but a crawler can ignore it. To see which bots really visit, look at your server or CDN logs.
: Yes. The generator runs in your browser and nothing you enter is sent to us. Copy the result or download it as robots.txt.
: Review it whenever you add pages you want to keep private and every few months, because AI companies launch new crawlers. This generator lists the AI bots we track.