Free tool

Block AI training, keep AI search. Or the other way round.

AI companies run separate bots for training, for search, and for fetching a page when someone asks. Pick what to allow for each kind and copy the rules into your robots.txt. It all runs in your browser; nothing is sent to us.

Training bots

Collects pages to train AI models.

Search bots

Builds the index an AI search feature answers from.

User-triggered bots

Fetches a page because a person asked something just now.

Your robots.txt block

# AI crawler rules from tryrenown.com/tools/robots-txt-generator
# Bot list version 2026-11-10.1

# Blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-CloudVertexBot
User-agent: GoogleOther
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: AI2Bot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Allowed bots are not listed: they follow your existing User-agent: * rules.

Add it to the robots.txt at the root of each host (www and non-www count separately), above or below your existing rules. Then run the crawler checker to confirm each bot reads it the way you meant.

What each choice costs

Blocking training bots doesn't take you out of AI search

If you block training bots

Blocking training bots asks the vendor to leave your pages out of future training data. It does not take you out of AI search. OpenAI says GPTBot and OAI-SearchBot are set independently, and Google says Google-Extended doesn't affect inclusion in Google Search. It also doesn't undo the past: Anthropic talks about excluding your future content.

If you block search bots

Blocking search bots is what takes you out of AI search answers. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude's results. Googlebot and Bingbot sit here too, and blocking those removes you from Google and Bing, not only their AI features.

If you block user-triggered bots

These bots fetch a page because someone asked about it a moment ago, so blocking them means the assistant can't read your page for that person. Vendors treat the rule differently. Anthropic says blocking Claude-User stops Claude retrieving your content for a user's query, and Mistral says robots.txt governs MistralAI-User. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity and Google say their user-triggered fetchers generally ignore robots.txt. For those, the line is a request at best.

robots.txt is a request, not a lock

Well-behaved bots read it and obey; others don't. The ai.robots.txt project, for example, records ByteDance's Bytespider as not respecting it. Changes also take time to land: OpenAI says about 24 hours for search, DuckDuckGo says 72 hours for DuckAssistBot, and Amazon says its bots may work from a copy of your robots.txt up to 30 days old. To enforce a block, you need a firewall rule, and to check who is really visiting, the vendor's published IP ranges. The crawler directory lists them.

What next

AI crawler directory

Look up a bot from your logs

Who runs it, what it collects, the user-agent token to use in robots.txt, and how to tell a real visit from one faking its user agent.

AI visibility

Find out whether AI recommends you

Being readable is only the start. The free AI visibility check asks AI the questions your customers ask, shows the answers word for word, and tells you whether you were named.