169 AI bots, sorted by what they want.
74 collect pages for training, 38 build indexes for AI search, and 57 fetch a page because a person asked a question just now. Only 27 can be checked against IP ranges the vendor publishes. For the rest, the user agent is all you have, and anyone can send any user agent.
Based on the ai.robots.txt list, checked against vendor documentation. List version 2026-11-10.1, updated 10 November 2026. The same list drives Renown's Crawler Insights.
Training
Collects pages to train AI models.
Search
Builds the index an AI search feature answers from.
User-triggered
Fetches a page because a person asked something just now.
| User agent | Vendor | Type | Verifiable |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search | IP ranges |
| ChatGPT-User | OpenAI | User-triggered | IP ranges |
| GPTBot | OpenAI | Training | IP ranges |
| ChatGPT Agent | OpenAI | User-triggered | No |
| Claude-SearchBot | Anthropic | Search | IP ranges |
| Claude-User | Anthropic | User-triggered | IP ranges |
| ClaudeBot | Anthropic | Training | IP ranges |
| Claude-Agent | Anthropic | User-triggered | No |
| Claude-Code | Anthropic | User-triggered | No |
| Claude-Web | Anthropic | Training | No |
| anthropic-ai | Anthropic | Training | No |
| Perplexity-User | Perplexity | User-triggered | IP ranges |
| PerplexityBot | Perplexity | Search | IP ranges |
| Google-CloudVertexBot | Training | IP ranges + rDNS | |
| Google-Agent | User-triggered | IP ranges | |
| Google-GeminiNotebook | User-triggered | IP ranges + rDNS | |
| GoogleAgent-URLContext | User-triggered | IP ranges + rDNS | |
| GoogleAgent-Mariner | User-triggered | IP ranges + rDNS | |
| Gemini-Deep-Research | User-triggered | IP ranges + rDNS | |
| Google-Firebase | User-triggered | IP ranges + rDNS | |
| Google-Gemini-CLI | User-triggered | No | |
| GoogleOther | Training | IP ranges + rDNS | |
| Googlebot | Search | IP ranges + rDNS | |
| bingbot | Microsoft | Search | IP ranges + rDNS |
| AzureAI-SearchBot | Microsoft | Search | No |
| Applebot | Apple | Search | IP ranges + rDNS |
| meta-externalfetcher | Meta | User-triggered | No |
| meta-webindexer | Meta | Search | No |
| meta-externalagent | Meta | Training | No |
| FacebookBot | Meta | Training | No |
| CCBot | Common Crawl | Training | IP ranges + rDNS |
| Bytespider | ByteDance | Training | No |
| TikTokSpider | ByteDance | Training | No |
| DoubaoBot | ByteDance | User-triggered | No |
| Amzn-SearchBot | Amazon | Search | IP ranges |
| Amzn-User | Amazon | User-triggered | IP ranges |
| Amazonbot | Amazon | Training | IP ranges |
| DuckAssistBot | DuckDuckGo | Search | IP ranges |
| DuckDuckBot | DuckDuckGo | Search | IP ranges |
| MistralAI-User | Mistral | User-triggered | IP ranges |
| MistralAI-Index | Mistral | Search | IP ranges |
| MistralAI-Training | Mistral | Training | No |
| AI2Bot-DeepResearchEval | Ai2 | User-triggered | No |
| Ai2Bot-Dolma | Ai2 | Training | No |
| AI2Bot | Ai2 | Training | No |
| AddSearchBot | Unknown | Search | No |
| AgentDataBot | Unknown | Training | No |
| AgentTimes | The Agent Times | Training | No |
| AGI Agent | AGI, Inc | User-triggered | No |
| aiHitBot | aiHit | Training | No |
| AIWebIndex | Lyrenth | Search | No |
| amazon-kendra | Amazon | Search | No |
| amazon-QBusiness | Amazon | User-triggered | No |
| AmazonBuyForMe | Amazon | User-triggered | No |
| Andibot | Andi | Search | No |
| Anomura | Direqt | Search | No |
| ApifyBot | Apify | Training | No |
| ApifyWebsiteContentCrawler | Apify | Training | No |
| Aranet-SearchBot | Unknown | User-triggered | No |
| atlassian-bot | Atlassian | User-triggered | No |
| Awario | Awario | Training | No |
| bedrockbot | Amazon | Training | No |
| bigsur.ai | Unknown | User-triggered | No |
| BixelBot | Unknown | Training | No |
| Bravebot | Brave | Search | No |
| Brightbot | Bright Data | Training | No |
| BuddyBot | BuddyBotLearning | Training | No |
| CartAI Agentic Commerce | Unknown | User-triggered | No |
| Channel3Bot | Unknown | Search | No |
| ChatGLM-Spider | Zhipu AI | Training | No |
| CledaraBot | Unknown | User-triggered | No |
| Cloudflare-AutoRAG | Cloudflare | Search | No |
| CloudflareBrowserRenderingCrawler | Cloudflare | Training | No |
| cohere-ai | Cohere | User-triggered | No |
| cohere-training-data-crawler | Cohere | Training | No |
| Cotoyogi | ROIS | Training | No |
| CragCrawler | Unknown | Training | No |
| Crawl4AI | Crawl4AI | User-triggered | No |
| Crawlspace | Crawlspace | Training | No |
| Datenbank Crawler | Datenbank | Training | No |
| DeepSeekBot | DeepSeek | Training | No |
| Devin | Cognition | User-triggered | No |
| Diffbot-User | Diffbot | User-triggered | No |
| Diffbot | Diffbot | Training | No |
| EchoboxBot | Echobox | Training | No |
| ElevenlabsBot | ElevenLabs | User-triggered | No |
| ElevenlabsCrawler | ElevenLabs | Training | No |
| ERNIEBot | Baidu | Training | No |
| ExaBot | Exa | Search | No |
| ExaSearchBot | Exa | Search | No |
| ExaUser | Exa | User-triggered | No |
| Factset_spyderbot | FactSet | Training | No |
| FirecrawlAgent | Firecrawl | Training | No |
| FriendlyCrawler | Unknown | Training | No |
| GeistHaus-PageFetcher | Unknown | User-triggered | No |
| GrokAppAndroid | xAI | Search | No |
| GrokBot | xAI | Training | No |
| HarkBot | Unknown | User-triggered | No |
| HenkBot | Unknown | Training | No |
| Hunyuan | Tencent | Training | No |
| iaskspider | iAsk | Search | No |
| ICC-Crawler | NICT | Training | No |
| ImagesiftBot | ImageSift | Training | No |
| imageSpider | Unknown | Training | No |
| img2dataset | img2dataset | User-triggered | No |
| ISSCyberRiskCrawler | ISS-Corporate | Training | No |
| kagi-fetcher | Kagi | User-triggered | No |
| Kangaroo Bot | Unknown | Training | No |
| KeenableBot | Keenable | Search | No |
| Kimi-Agent | Moonshot AI | User-triggered | No |
| Kimi-SearchBot | Moonshot AI | Search | No |
| Kimi-User | Moonshot AI | User-triggered | No |
| KimiBot | Moonshot AI | Training | No |
| KlaviyoAIBot | Klaviyo | Search | No |
| KunatoCrawler | Unknown | User-triggered | No |
| laion-huggingface-processor | Unknown | Training | No |
| LAIONDownloader | Large-scale Artificial Intelligence Open Network | Search | No |
| Lightpanda | Lightpanda | User-triggered | No |
| LinerBot | Unknown | User-triggered | No |
| Linguee Bot | DeepL/Linguee | Training | No |
| LinkupBot | Linkup | Search | No |
| Manus-User | Manus | User-triggered | No |
| MoonshotBot | Moonshot AI | Training | No |
| Mozilla-Tabstack | Mozilla | User-triggered | No |
| MyCentralAIScraperBot | Unknown | Training | No |
| NagetBot | Unknown | Training | No |
| netEstate Imprint Crawler | netEstate | Training | No |
| newsai | Unknown | Training | No |
| NovaAct | Amazon | User-triggered | No |
| omgilibot | Webz.io | Training | No |
| omgili | Webz.io | Training | No |
| PanguBot | Huawei | Training | No |
| Panscient | Panscient | Training | No |
| Payhawk Invoice Fetching Agent | Unknown | User-triggered | No |
| PetalBot | Huawei | Search | No |
| PhindBot | Phind | Search | No |
| Poggio-Citations | Unknown | User-triggered | No |
| Poseidon Research Crawler | Poseidon Research | Search | No |
| qodercli | Qoder | User-triggered | No |
| QualifiedBot | Qualified | User-triggered | No |
| Querit-SearchBot | Querit | Search | No |
| QueritBot | Querit | Search | No |
| QuillBot | QuillBot | Training | No |
| QwenBot | Alibaba | Training | No |
| Reflectionbot | Reflection | User-triggered | No |
| SBIntuitionsBot | SB Intuitions | Training | No |
| ScryBot | Scry | User-triggered | No |
| SemrushBot-OCOB | Semrush | Training | No |
| SemrushBot-SWA | Semrush | User-triggered | No |
| Shap-User | Parallel | User-triggered | No |
| ShapBot | Parallel | Training | No |
| Sidetrade indexer bot | Sidetrade | Training | No |
| TavilyBot | Tavily | Search | No |
| TerraCotta | Ceramic AI | Training | No |
| Thinkbot | Thinkbot | Training | No |
| Timpibot | Timpi | Training | No |
| TongyiBot | Alibaba | User-triggered | No |
| TwinAgent | Unknown | User-triggered | No |
| UseAI | Unknown | User-triggered | No |
| VelenPublicWebCrawler | Velen Crawler | Training | No |
| WARDBot | WEBSPARK | Training | No |
| wpbot | QuantumCloud | Training | No |
| WRTNBot | Unknown | User-triggered | No |
| xAI-Grok | xAI | Training | No |
| YandexAdditionalBot | Yandex | Training | No |
| YiBot | 01.ai | Search | No |
| YiyanBot | Baidu | User-triggered | No |
| YouBot | You.com | Search | No |
| ZanistaBot | Unknown | Search | No |
A page for each of the 28 you'll meet most
What the vendor says each bot is for, what it says about robots.txt, and how to check a hit is real.
- OAI-SearchBotOpenAI · Search
- ChatGPT-UserOpenAI · User-triggered
- GPTBotOpenAI · Training
- Claude-SearchBotAnthropic · Search
- Claude-UserAnthropic · User-triggered
- ClaudeBotAnthropic · Training
- Perplexity-UserPerplexity · User-triggered
- PerplexityBotPerplexity · Search
- Google-CloudVertexBotGoogle · Training
- Google-AgentGoogle · User-triggered
- Google-GeminiNotebookGoogle · User-triggered
- GoogleOtherGoogle · Training
- GooglebotGoogle · Search
- BingbotMicrosoft · Search
- ApplebotApple · Search
- Meta-ExternalFetcherMeta · User-triggered
- Meta-WebIndexerMeta · Search
- Meta-ExternalAgentMeta · Training
- CCBotCommon Crawl · Training
- BytespiderByteDance · Training
- Amzn-SearchBotAmazon · Search
- Amzn-UserAmazon · User-triggered
- AmazonbotAmazon · Training
- DuckAssistBotDuckDuckGo · Search
- MistralAI-UserMistral · User-triggered
- MistralAI-IndexMistral · Search
- MistralAI-TrainingMistral · Training
- AI2BotAi2 · Training
Decide what to allow with the robots.txt generator, or check an address from your logs with the IP verifier.
AI crawler checker
See what AI bots get from your site
Enter a URL and see, bot by bot, whether robots.txt lets it in, whether a firewall turns it away, and how much of the page it can read without running JavaScript.
AI visibility
Find out whether AI recommends you
Being readable is only the start. The free AI visibility check asks AI the questions your customers ask, shows the answers word for word, and tells you whether you were named.