AI Robots.txt Generator
The AI Robots.txt Generator is a free browser tool that writes a robots.txt for the AI bots we track. You choose which ones may read your site, copy the result or download it, and upload it to the root of your domain.
To check a robots.txt that is already live, use the Robots.txt Checker. Both sit with the rest of our free tools at Free AI Search Tools.
The three kinds of AI bots
The generator groups every bot by what it does, because blocking each group has a different cost.
- AI search crawlers (for example OAI-SearchBot, PerplexityBot and Claude-SearchBot) index pages so AI search engines can cite and link to them. Block them and AI search stops citing your pages.
- User-triggered AI assistants (for example ChatGPT-User, Perplexity-User and Claude-User) open a page when a person asks an assistant to read it. Block them and assistants cannot open your pages for those people.
- AI training crawlers (for example GPTBot, ClaudeBot, Google-Extended and CCBot) collect pages to train models. Block them and future models learn less about you, while AI search can still cite you.
How to use it
- Pick a preset: Allow all AI bots, Block training only or Block all AI bots.
- Switch any single bot on or off. Each switch shows the company that runs the bot.
- Optionally list private paths, one per line. Every bot you allow skips those paths, and so does every other crawler.
- Optionally add your sitemap URL.
- Copy the file or download it as robots.txt, then upload it to the root of your domain, for example example.com/robots.txt.
Blocked bots are grouped under one rule that disallows the whole site. Allowed bots get their own group with your private paths, because a bot that has a group of its own ignores the rules written for every other crawler.
What robots.txt cannot do
Robots.txt is a request, not a lock. Well-known AI companies say their bots follow it, but a crawler can ignore it. Blocking a training crawler stops new pages from entering future training data; it does not remove what a model already learned. To see which bots really visit your site, connect your server or CDN logs in Agent Analytics.
Privacy
The generator runs in your browser. Nothing you enter is sent to us.
