Meta
Meta-ExternalAgent
Fetching public pages for Meta's training, as documented separately from the crawler that loads link previews.
- Who runs it
- Meta
- robots.txt token
- meta-externalagent
- User-agent
- meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
Meta's crawler documentation is the source for the current product token and user-agent. FacebookExternalHit, which fetches link previews, is a different agent.
Meta documents Meta-ExternalAgent as a crawler that collects public web content. The robots.txt token is meta-externalagent.
Link previews in Facebook use a different crawler. Blocking Meta-ExternalAgent is not the same switch as blocking link unfurls. Read Meta's crawler page before you collapse them into one rule.
The token does not affect Googlebot.
Example robots.txt rules
Examples for your file, using the documented token. They are not a copy of the operator's own robots.txt.
Allow
User-agent: meta-externalagent Allow: /
Block
User-agent: meta-externalagent Disallow: /
Related guides
Other crawlers
Questions
- Will this stop Facebook from showing a preview of my link?
- No. Previews are fetched by a different agent. Meta's crawler doc names both.