Free · Runs in your browser · Nothing is sent anywhere
robots.txt tester for AI bots
Paste a robots.txt and a URL. See, for 30 AI and search bots, whether each one may fetch it and which line made the call. It follows RFC 9309, the same rules the crawlers use.
testing /account/settings
| Bot | Result |
|---|---|
| OAI-SearchBotOpenAI · search | allowedline 12 |
| ChatGPT-UserOpenAI · user-triggered | blockedline 4 · via * |
| GPTBotOpenAI · training | blockedline 9 |
| Claude-SearchBotAnthropic · search | blockedline 4 · via * |
| Claude-UserAnthropic · user-triggered | blockedline 4 · via * |
| ClaudeBotAnthropic · training | blockedline 4 · via * |
| Perplexity-UserPerplexity · user-triggered | blockedline 4 · via * |
| PerplexityBotPerplexity · search | blockedline 4 · via * |
| Google-CloudVertexBotGoogle · training | blockedline 4 · via * |
| Google-AgentGoogle · user-triggered | blockedline 4 · via * |
| Google-GeminiNotebookGoogle · user-triggered | blockedline 4 · via * |
| GoogleOtherGoogle · training | blockedline 4 · via * |
| GooglebotGoogle · search | blockedline 4 · via * |
| bingbotMicrosoft · search | blockedline 4 · via * |
| ApplebotApple · search | blockedline 4 · via * |
| meta-externalfetcherMeta · user-triggered | blockedline 4 · via * |
| meta-webindexerMeta · search | blockedline 4 · via * |
| meta-externalagentMeta · training | blockedline 4 · via * |
| CCBotCommon Crawl · training | blockedline 9 |
| BytespiderByteDance · training | blockedline 4 · via * |
| Amzn-SearchBotAmazon · search | blockedline 4 · via * |
| Amzn-UserAmazon · user-triggered | blockedline 4 · via * |
| AmazonbotAmazon · training | blockedline 4 · via * |
| DuckAssistBotDuckDuckGo · search | blockedline 4 · via * |
| MistralAI-UserMistral · user-triggered | blockedline 4 · via * |
| MistralAI-IndexMistral · search | blockedline 4 · via * |
| MistralAI-TrainingMistral · training | blockedline 4 · via * |
| AI2BotAi2 · training | blockedline 4 · via * |
| Google-ExtendedGoogle · training | allowedno rule |
| Applebot-ExtendedApple · training | blockedline 4 · via * |
- line 15 "disalow" is not a robots.txt field. Check the spelling.
- Sitemap: https://www.example.com/sitemap.xml
Your browser can't read another site's robots.txt, so paste it in. To fetch it live and test your firewall too, use the AI crawler checker.
Groups don't add up
The mistake we see most: a file keeps everyone out of /account/ under User-agent: *, then adds a group for OAI-SearchBot to let it in. That bot now ignores the * group completely, and it can fetch the account pages too.
A crawler picks the one group that names it and obeys only that. If you want the shared rules too, copy them into the named group. The sample shows it: test /account/settings and OAI-SearchBot is the one bot let through. Google-Extended gets in as well, because its rule is spelled Disalow.
Longest match wins, and Allow wins a tie
Order in the file doesn't matter. Inside a group, the rule with the longest matching path decides. So Allow: /account/help beats Disallow: /account/ for the help page, wherever each line sits. When an Allow and a Disallow are the same length, Allow wins.
* matches anything and $ pins the end of the URL. Disallow: /*.pdf$ blocks PDFs and nothing else. Want a file written for you instead? The robots.txt generator does that.
Questions
If I block GPTBot, am I out of ChatGPT?
No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches a page when someone asks about it. Each one reads its own rules, so you can block training and stay in search. Test all three here.
Why does a bot I didn't mention show as blocked?
Bots with no group of their own follow the User-agent: * group. If that group disallows a path, every unnamed bot is kept out of it, AI or not.
Why does a bot ignore my User-agent: * rules?
Because it has a group of its own. A crawler obeys only the most specific group that names it, and the * rules no longer apply to it. If you want them, repeat them inside the named group.
Can this fetch my live robots.txt?
Not from your browser, because most sites don't allow other pages to read their files. Open yoursite.com/robots.txt, copy it in, or use the AI crawler checker, which fetches it from our server and also tests your firewall.
Do all bots respect robots.txt?
The big vendors say their crawlers do. Several say their user-triggered fetchers may not, since a person asked for the page. This tool shows what the file says. It can't make anyone obey it.
More free tools
These tools check one thing at a time. Renown checks what AI says about you, every two weeks.
It asks ChatGPT, Claude, Gemini and Perplexity the questions your customers ask, shows you the answers, and lists what to fix when you're not named. The tools stay free. Renown is how we pay for them.