Meta
Meta-ExternalAgent is Meta's AI training crawler used to collect data for Llama models and Meta AI products like Meta AI on WhatsApp, Instagram, and Facebook.
Current Rank
#4
Traffic Share
15.29%
Operator
Meta
Category
AI Training Crawler
Meta-ExternalAgent is a web crawler operated by Meta (formerly Facebook) to collect training data for their AI models, including the Llama family of large language models.
This crawler is separate from facebookexternalhit, which is Meta's link preview crawler. Meta-ExternalAgent specifically gathers data for AI training purposes, supporting Meta AI products across WhatsApp, Instagram, Facebook, and Messenger.
Meta-ExternalAgent was introduced as Meta ramped up its AI investments. It crawls publicly available web content to train and improve Meta's AI capabilities.
To block Meta-ExternalAgent from crawling your site for AI training purposes, add the following to your robots.txt:
robots.txt
User-agent: Meta-ExternalAgent Disallow: /
Blocking Meta-ExternalAgent only prevents AI training crawling. It does not affect how your links appear when shared on Facebook or Instagram — that is handled by facebookexternalhit, which is a separate crawler.
User-Agent string: Meta-ExternalAgent/1.0 (+https://www.facebook.com/externalhit_uatext.php)
Respects robots.txt directives
Separate from facebookexternalhit (link preview crawler)
Used for training Llama and other Meta AI models
One of the fastest-growing AI crawlers by traffic volume
Allowing Meta-ExternalAgent to crawl your site means your content may be used to train Meta's Llama models and other AI products. This can influence how Meta AI responds to questions about your brand across WhatsApp, Instagram, Facebook, and Messenger.
Meta AI is used by billions of people globally. Having accurate AI training data from your site could improve how Meta AI represents your brand in conversations.
LLM Pulse monitors how AI models mention your brand. See which crawlers visit your site and how that translates into AI visibility.