Bytespider is ByteDance's training crawler.
ByteDance's crawler. ByteDance publishes no crawler documentation in our sources; the ai.robots.txt project lists its function as LLM training.
Source: ai.robots.txt project- Vendor
- ByteDance
- Type
- Training. Collects pages to train AI models.
- User agent contains
- Bytespider
- robots.txt token
- Bytespider
- Published IP ranges
- None published
The ai.robots.txt project records Bytespider as not respecting robots.txt.
To block Bytespider:
User-agent: Bytespider Disallow: /
To allow it explicitly:
User-agent: Bytespider Allow: /
Is that request really Bytespider?
ByteDance publishes no IP ranges or reverse-DNS check for Bytespider in the sources our list uses. A request with this user agent can't be told apart from a scraper borrowing it. If the traffic is a problem, a firewall rule on the user agent is the only lever, and it will catch impostors too.
From Renown's bot list, updated 10 November 2026. See all 169
AI crawler checker
See what AI bots get from your site
Enter a URL and see, bot by bot, whether robots.txt lets it in, whether a firewall turns it away, and how much of the page it can read without running JavaScript.
AI visibility
Find out whether AI recommends you
Being readable is only the start. The free AI visibility check asks AI the questions your customers ask, shows the answers word for word, and tells you whether you were named.
Questions people ask
What is Bytespider?
Bytespider is ByteDance's training crawler. ByteDance's crawler. ByteDance publishes no crawler documentation in our sources; the ai.robots.txt project lists its function as LLM training.
Does Bytespider follow robots.txt?
The ai.robots.txt project records Bytespider as not respecting robots.txt.
How do I check a Bytespider request is real?
You can't, from the outside. ByteDance publishes no IP ranges or reverse-DNS check for Bytespider in the sources our list uses, so a request with this user agent may come from anyone.