Block AI training, keep AI search. Or the other way round.
AI companies run separate bots for training, for search, and for fetching a page when someone asks. Pick what to allow for each kind and copy the rules into your robots.txt. It all runs in your browser; nothing is sent to us.
Training bots
Collects pages to train AI models.
Search bots
Builds the index an AI search feature answers from.
User-triggered bots
Fetches a page because a person asked something just now.
Your robots.txt block
# AI crawler rules from tryrenown.com/tools/robots-txt-generator # Bot list version 2026-11-10.1 # Blocked User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: meta-externalagent User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot User-agent: MistralAI-Training User-agent: AI2Bot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: / # Allowed bots are not listed: they follow your existing User-agent: * rules.
Add it to the robots.txt at the root of each host (www and non-www count separately), above or below your existing rules. Then run the crawler checker to confirm each bot reads it the way you meant.
Blocking training bots doesn't take you out of AI search
If you block training bots
Blocking training bots asks the vendor to leave your pages out of future training data. It does not take you out of AI search. OpenAI says GPTBot and OAI-SearchBot are set independently, and Google says Google-Extended doesn't affect inclusion in Google Search. It also doesn't undo the past: Anthropic talks about excluding your future content.
If you block search bots
Blocking search bots is what takes you out of AI search answers. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude's results. Googlebot and Bingbot sit here too, and blocking those removes you from Google and Bing, not only their AI features.
If you block user-triggered bots
These bots fetch a page because someone asked about it a moment ago, so blocking them means the assistant can't read your page for that person. Vendors treat the rule differently. Anthropic says blocking Claude-User stops Claude retrieving your content for a user's query, and Mistral says robots.txt governs MistralAI-User. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity and Google say their user-triggered fetchers generally ignore robots.txt. For those, the line is a request at best.
robots.txt is a request, not a lock
Well-behaved bots read it and obey; others don't. The ai.robots.txt project, for example, records ByteDance's Bytespider as not respecting it. Changes also take time to land: OpenAI says about 24 hours for search, DuckDuckGo says 72 hours for DuckAssistBot, and Amazon says its bots may work from a copy of your robots.txt up to 30 days old. To enforce a block, you need a firewall rule, and to check who is really visiting, the vendor's published IP ranges. The crawler directory lists them.
AI crawler directory
Look up a bot from your logs
Who runs it, what it collects, the user-agent token to use in robots.txt, and how to tell a real visit from one faking its user agent.
AI visibility
Find out whether AI recommends you
Being readable is only the start. The free AI visibility check asks AI the questions your customers ask, shows the answers word for word, and tells you whether you were named.