Which AI bots can read your site, and which get turned away.
Enter a domain. We read its robots.txt for every well-known AI bot, fetch the homepage as a browser and as a few of the AI bots that matter most, and show what each one gets. Results appear on this page. No email.
Four ways a site goes missing from AI answers
- robots.txt. A rule for GPTBot only affects training. A rule for OAI-SearchBot decides whether you can appear in ChatGPT search, and PerplexityBot is the bot that puts you in Perplexity's results. A broad
User-agent: *block catches every bot that follows the standard. - Firewalls and CDNs. Bot protection can return 403 or a challenge page to AI bots while browsers get the site. Nobody gets an email when that happens.
- JavaScript. If the text only appears after scripts run, bots that read raw HTML see an empty page.
- Missing basics. No sitemap, no structured data, no page title. Small things, and cheap to fix.
Want to change what you allow? The robots.txt generator explains each choice.
Questions people ask
If I block GPTBot, do I disappear from ChatGPT?
No. GPTBot collects pages that may be used for training. OAI-SearchBot is the bot behind ChatGPT search, and OpenAI says sites that opt out of it are not shown in ChatGPT search answers. OpenAI also says each setting is independent of the others.
Why does a bot show as blocked when robots.txt allows it?
Something in front of the site, usually a firewall, a CDN's bot protection or a security plugin, answered the bot's user agent with an error or a challenge page while a browser got the real page. robots.txt can't override that. Check your CDN and security settings for AI bot rules.
Does this show which bots actually visit my site?
No. It shows what a bot would get if it came. Real visits are in your server or CDN access logs: search them for the bot's user-agent token, then check the IP with the AI bot IP verifier, because some scrapers borrow a well-known bot's name.
Do you store the results?
We cache the result for each domain for an hour so repeat checks don't hit your site again. We don't ask for an email and the check doesn't create an account.