AI crawler directory

169 AI bots, sorted by what they want.

74 collect pages for training, 38 build indexes for AI search, and 57 fetch a page because a person asked a question just now. Only 27 can be checked against IP ranges the vendor publishes. For the rest, the user agent is all you have, and anyone can send any user agent.

Based on the ai.robots.txt list, checked against vendor documentation. List version 2026-11-10.1, updated 10 November 2026. The same list drives Renown's Crawler Insights.

Training

Collects pages to train AI models.

Search

Builds the index an AI search feature answers from.

User-triggered

Fetches a page because a person asked something just now.

User agentVendorTypeVerifiable
OAI-SearchBotOpenAISearchIP ranges
ChatGPT-UserOpenAIUser-triggeredIP ranges
GPTBotOpenAITrainingIP ranges
ChatGPT AgentOpenAIUser-triggeredNo
Claude-SearchBotAnthropicSearchIP ranges
Claude-UserAnthropicUser-triggeredIP ranges
ClaudeBotAnthropicTrainingIP ranges
Claude-AgentAnthropicUser-triggeredNo
Claude-CodeAnthropicUser-triggeredNo
Claude-WebAnthropicTrainingNo
anthropic-aiAnthropicTrainingNo
Perplexity-UserPerplexityUser-triggeredIP ranges
PerplexityBotPerplexitySearchIP ranges
Google-CloudVertexBotGoogleTrainingIP ranges + rDNS
Google-AgentGoogleUser-triggeredIP ranges
Google-GeminiNotebookGoogleUser-triggeredIP ranges + rDNS
GoogleAgent-URLContextGoogleUser-triggeredIP ranges + rDNS
GoogleAgent-MarinerGoogleUser-triggeredIP ranges + rDNS
Gemini-Deep-ResearchGoogleUser-triggeredIP ranges + rDNS
Google-FirebaseGoogleUser-triggeredIP ranges + rDNS
Google-Gemini-CLIGoogleUser-triggeredNo
GoogleOtherGoogleTrainingIP ranges + rDNS
GooglebotGoogleSearchIP ranges + rDNS
bingbotMicrosoftSearchIP ranges + rDNS
AzureAI-SearchBotMicrosoftSearchNo
ApplebotAppleSearchIP ranges + rDNS
meta-externalfetcherMetaUser-triggeredNo
meta-webindexerMetaSearchNo
meta-externalagentMetaTrainingNo
FacebookBotMetaTrainingNo
CCBotCommon CrawlTrainingIP ranges + rDNS
BytespiderByteDanceTrainingNo
TikTokSpiderByteDanceTrainingNo
DoubaoBotByteDanceUser-triggeredNo
Amzn-SearchBotAmazonSearchIP ranges
Amzn-UserAmazonUser-triggeredIP ranges
AmazonbotAmazonTrainingIP ranges
DuckAssistBotDuckDuckGoSearchIP ranges
DuckDuckBotDuckDuckGoSearchIP ranges
MistralAI-UserMistralUser-triggeredIP ranges
MistralAI-IndexMistralSearchIP ranges
MistralAI-TrainingMistralTrainingNo
AI2Bot-DeepResearchEvalAi2User-triggeredNo
Ai2Bot-DolmaAi2TrainingNo
AI2BotAi2TrainingNo
AddSearchBotUnknownSearchNo
AgentDataBotUnknownTrainingNo
AgentTimesThe Agent TimesTrainingNo
AGI AgentAGI, IncUser-triggeredNo
aiHitBotaiHitTrainingNo
AIWebIndexLyrenthSearchNo
amazon-kendraAmazonSearchNo
amazon-QBusinessAmazonUser-triggeredNo
AmazonBuyForMeAmazonUser-triggeredNo
AndibotAndiSearchNo
AnomuraDireqtSearchNo
ApifyBotApifyTrainingNo
ApifyWebsiteContentCrawlerApifyTrainingNo
Aranet-SearchBotUnknownUser-triggeredNo
atlassian-botAtlassianUser-triggeredNo
AwarioAwarioTrainingNo
bedrockbotAmazonTrainingNo
bigsur.aiUnknownUser-triggeredNo
BixelBotUnknownTrainingNo
BravebotBraveSearchNo
BrightbotBright DataTrainingNo
BuddyBotBuddyBotLearningTrainingNo
CartAI Agentic CommerceUnknownUser-triggeredNo
Channel3BotUnknownSearchNo
ChatGLM-SpiderZhipu AITrainingNo
CledaraBotUnknownUser-triggeredNo
Cloudflare-AutoRAGCloudflareSearchNo
CloudflareBrowserRenderingCrawlerCloudflareTrainingNo
cohere-aiCohereUser-triggeredNo
cohere-training-data-crawlerCohereTrainingNo
CotoyogiROISTrainingNo
CragCrawlerUnknownTrainingNo
Crawl4AICrawl4AIUser-triggeredNo
CrawlspaceCrawlspaceTrainingNo
Datenbank CrawlerDatenbankTrainingNo
DeepSeekBotDeepSeekTrainingNo
DevinCognitionUser-triggeredNo
Diffbot-UserDiffbotUser-triggeredNo
DiffbotDiffbotTrainingNo
EchoboxBotEchoboxTrainingNo
ElevenlabsBotElevenLabsUser-triggeredNo
ElevenlabsCrawlerElevenLabsTrainingNo
ERNIEBotBaiduTrainingNo
ExaBotExaSearchNo
ExaSearchBotExaSearchNo
ExaUserExaUser-triggeredNo
Factset_spyderbotFactSetTrainingNo
FirecrawlAgentFirecrawlTrainingNo
FriendlyCrawlerUnknownTrainingNo
GeistHaus-PageFetcherUnknownUser-triggeredNo
GrokAppAndroidxAISearchNo
GrokBotxAITrainingNo
HarkBotUnknownUser-triggeredNo
HenkBotUnknownTrainingNo
HunyuanTencentTrainingNo
iaskspideriAskSearchNo
ICC-CrawlerNICTTrainingNo
ImagesiftBotImageSiftTrainingNo
imageSpiderUnknownTrainingNo
img2datasetimg2datasetUser-triggeredNo
ISSCyberRiskCrawlerISS-CorporateTrainingNo
kagi-fetcherKagiUser-triggeredNo
Kangaroo BotUnknownTrainingNo
KeenableBotKeenableSearchNo
Kimi-AgentMoonshot AIUser-triggeredNo
Kimi-SearchBotMoonshot AISearchNo
Kimi-UserMoonshot AIUser-triggeredNo
KimiBotMoonshot AITrainingNo
KlaviyoAIBotKlaviyoSearchNo
KunatoCrawlerUnknownUser-triggeredNo
laion-huggingface-processorUnknownTrainingNo
LAIONDownloaderLarge-scale Artificial Intelligence Open NetworkSearchNo
LightpandaLightpandaUser-triggeredNo
LinerBotUnknownUser-triggeredNo
Linguee BotDeepL/LingueeTrainingNo
LinkupBotLinkupSearchNo
Manus-UserManusUser-triggeredNo
MoonshotBotMoonshot AITrainingNo
Mozilla-TabstackMozillaUser-triggeredNo
MyCentralAIScraperBotUnknownTrainingNo
NagetBotUnknownTrainingNo
netEstate Imprint CrawlernetEstateTrainingNo
newsaiUnknownTrainingNo
NovaActAmazonUser-triggeredNo
omgilibotWebz.ioTrainingNo
omgiliWebz.ioTrainingNo
PanguBotHuaweiTrainingNo
PanscientPanscientTrainingNo
Payhawk Invoice Fetching AgentUnknownUser-triggeredNo
PetalBotHuaweiSearchNo
PhindBotPhindSearchNo
Poggio-CitationsUnknownUser-triggeredNo
Poseidon Research CrawlerPoseidon ResearchSearchNo
qodercliQoderUser-triggeredNo
QualifiedBotQualifiedUser-triggeredNo
Querit-SearchBotQueritSearchNo
QueritBotQueritSearchNo
QuillBotQuillBotTrainingNo
QwenBotAlibabaTrainingNo
ReflectionbotReflectionUser-triggeredNo
SBIntuitionsBotSB IntuitionsTrainingNo
ScryBotScryUser-triggeredNo
SemrushBot-OCOBSemrushTrainingNo
SemrushBot-SWASemrushUser-triggeredNo
Shap-UserParallelUser-triggeredNo
ShapBotParallelTrainingNo
Sidetrade indexer botSidetradeTrainingNo
TavilyBotTavilySearchNo
TerraCottaCeramic AITrainingNo
ThinkbotThinkbotTrainingNo
TimpibotTimpiTrainingNo
TongyiBotAlibabaUser-triggeredNo
TwinAgentUnknownUser-triggeredNo
UseAIUnknownUser-triggeredNo
VelenPublicWebCrawlerVelen CrawlerTrainingNo
WARDBotWEBSPARKTrainingNo
wpbotQuantumCloudTrainingNo
WRTNBotUnknownUser-triggeredNo
xAI-GrokxAITrainingNo
YandexAdditionalBotYandexTrainingNo
YiBot01.aiSearchNo
YiyanBotBaiduUser-triggeredNo
YouBotYou.comSearchNo
ZanistaBotUnknownSearchNo
What next

AI crawler checker

See what AI bots get from your site

Enter a URL and see, bot by bot, whether robots.txt lets it in, whether a firewall turns it away, and how much of the page it can read without running JavaScript.

AI visibility

Find out whether AI recommends you

Being readable is only the start. The free AI visibility check asks AI the questions your customers ask, shows the answers word for word, and tells you whether you were named.