Common Crawl · Training

CCBot is Common Crawl's training crawler.

Common Crawl's crawler, which builds an open repository of web crawl data.

Source: Common Crawl: CCBot
Facts
Vendor
Common Crawl
Type
Training. Collects pages to train AI models.
User agent contains
CCBot
robots.txt token
CCBot
Reverse DNS ends in
.crawl.commoncrawl.org

Note from our list: rDNS is IPv4 only.

robots.txt

Common Crawl's documented opt-out is a robots.txt group for CCBot with Disallow: /. It also warns that other crawlers falsely identify themselves as CCBot.

To block CCBot:

User-agent: CCBot
Disallow: /

To allow it explicitly:

User-agent: CCBot
Allow: /
Verify a hit

Is that request really CCBot?

  1. Take the client IP from your server or CDN log, not from a header the client can set.
  2. Check it against Common Crawl's published range listed above, or paste it into the AI bot IP verifier.
  3. Or run a reverse-DNS lookup on the IP. The hostname should end in .crawl.commoncrawl.org. Then look that hostname up and confirm it points back to the same IP.
  4. A CCBot user agent from an address that fails these checks is someone else using the name.

From Renown's bot list, updated 10 November 2026. See all 169

What next

AI crawler checker

See what AI bots get from your site

Enter a URL and see, bot by bot, whether robots.txt lets it in, whether a firewall turns it away, and how much of the page it can read without running JavaScript.

AI visibility

Find out whether AI recommends you

Being readable is only the start. The free AI visibility check asks AI the questions your customers ask, shows the answers word for word, and tells you whether you were named.

FAQ

Questions people ask

What is CCBot?

CCBot is Common Crawl's training crawler. Common Crawl's crawler, which builds an open repository of web crawl data.

Does CCBot follow robots.txt?

Common Crawl's documented opt-out is a robots.txt group for CCBot with Disallow: /. It also warns that other crawlers falsely identify themselves as CCBot.

How do I check a CCBot request is real?

Check the request's IP against Common Crawl's published ranges, or do a reverse-DNS lookup and confirm the hostname ends in .crawl.commoncrawl.org and resolves back to the same IP.