Googlebot is Google's search crawler.
Google Search's crawler. Google says crawling preferences addressed to Googlebot affect Google Search.
Source: Google: common crawlers- Vendor
- Type
- Search. Builds the index an AI search feature answers from.
- User agent contains
- Googlebot
- robots.txt token
- Googlebot
- Published IP ranges
- Reverse DNS ends in
- .googlebot.com, .google.com
Note from our list: Not AI-specific; included because it feeds AI Overviews/AI Mode. Substring also matches Googlebot-Image/-Video/-News. Google-Extended is a robots.txt token only.
Google-Extended is a separate robots.txt token with no crawler behind it. It controls whether content Google crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect a site's inclusion in Google Search and is not a ranking signal.
To block Googlebot:
User-agent: Googlebot Disallow: /
To allow it explicitly:
User-agent: Googlebot Allow: /
Is that request really Googlebot?
- Take the client IP from your server or CDN log, not from a header the client can set.
- Check it against Google's published range listed above, or paste it into the AI bot IP verifier.
- Or run a reverse-DNS lookup on the IP. The hostname should end in .googlebot.com or .google.com. Then look that hostname up and confirm it points back to the same IP.
- A Googlebot user agent from an address that fails these checks is someone else using the name.
Other Google bots
From Renown's bot list, updated 10 November 2026. See all 169
AI crawler checker
See what AI bots get from your site
Enter a URL and see, bot by bot, whether robots.txt lets it in, whether a firewall turns it away, and how much of the page it can read without running JavaScript.
AI visibility
Find out whether AI recommends you
Being readable is only the start. The free AI visibility check asks AI the questions your customers ask, shows the answers word for word, and tells you whether you were named.
Questions people ask
What is Googlebot?
Googlebot is Google's search crawler. Google Search's crawler. Google says crawling preferences addressed to Googlebot affect Google Search.
Does Googlebot follow robots.txt?
Google-Extended is a separate robots.txt token with no crawler behind it. It controls whether content Google crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect a site's inclusion in Google Search and is not a ranking signal.
How do I check a Googlebot request is real?
Check the request's IP against Google's published ranges, or do a reverse-DNS lookup and confirm the hostname ends in .googlebot.com or .google.com and resolves back to the same IP.