Meta · Search

Meta-WebIndexer is Meta's search crawler.

Crawls the web to improve the quality of Meta AI's search results.

Source: Meta web crawlers
Facts
Vendor
Meta
Type
Search. Builds the index an AI search feature answers from.
User agent contains
meta-webindexer
robots.txt token
meta-webindexer
Published IP ranges
None published
robots.txt

Meta says to block it by adding a disallow rule for it to robots.txt.

To block Meta-WebIndexer:

User-agent: meta-webindexer
Disallow: /

To allow it explicitly:

User-agent: meta-webindexer
Allow: /
Verify a hit

Is that request really Meta-WebIndexer?

Meta publishes no IP ranges or reverse-DNS check for Meta-WebIndexer in the sources our list uses. A request with this user agent can't be told apart from a scraper borrowing it. If the traffic is a problem, a firewall rule on the user agent is the only lever, and it will catch impostors too.

From Renown's bot list, updated 10 November 2026. See all 169

What next

AI crawler checker

See what AI bots get from your site

Enter a URL and see, bot by bot, whether robots.txt lets it in, whether a firewall turns it away, and how much of the page it can read without running JavaScript.

AI visibility

Find out whether AI recommends you

Being readable is only the start. The free AI visibility check asks AI the questions your customers ask, shows the answers word for word, and tells you whether you were named.

FAQ

Questions people ask

What is Meta-WebIndexer?

Meta-WebIndexer is Meta's search crawler. Crawls the web to improve the quality of Meta AI's search results.

Does Meta-WebIndexer follow robots.txt?

Meta says to block it by adding a disallow rule for it to robots.txt.

How do I check a Meta-WebIndexer request is real?

You can't, from the outside. Meta publishes no IP ranges or reverse-DNS check for Meta-WebIndexer in the sources our list uses, so a request with this user agent may come from anyone.