Agent Analytics (AI Bot Crawler Tracking)

Before an AI model can cite your site, its crawlers have to visit it. Agent Analytics is server-side AI bot traffic monitoring: it shows which AI crawlers (GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot, Google-Extended, and many more) are hitting your website and how that evolves over time.

With Agent Analytics, you can:

• See which AI bots (GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot, Google-Extended) crawl your site and how often
• Confirm that AI models are actually discovering new or updated pages before you expect them in AI answers
• Compare crawl volume by company (OpenAI, Anthropic, Google, Perplexity, and more) to see who is investing in your content
• Catch a drop in AI crawler activity that could explain falling citations

Find it under Bots & Traffic > Agent Analytics in the sidebar. Available on Scale plans and above.

In an embedded portal, Agent Analytics is read-only. Existing charts and tables are available, but connecting a provider, uploading logs, syncing, disconnecting, and managing a source must be handled in the main workspace. If the project is not connected yet, the embedded page asks you to contact your CSM.

What it does

• Tracks over 40 AI bots grouped by company (OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Microsoft, Amazon, and others)
• Daily traffic charts per bot and per company, so you can see whether AI crawlers are discovering your content
• Multiple ways to get your data in:
  • Cloudflare integration: paste a scoped API token (Zone Analytics: Read), pick a zone, and data syncs daily. Works on every Cloudflare plan, including Free.
  • Bunny CDN integration: paste your account API key and pick a pull zone; data syncs daily.
  • Log drains (real time): Vercel, Akamai DataStream 2, Fastly, Netlify, and GCP Load Balancer. Generate a unique ingest URL in the app and paste it into your provider's log drain settings.
  • Amazon S3 bucket: point us at a bucket of server access logs and we pull new files once a day (plus a manual "Sync now"). Works with S3-compatible stores such as Cloudflare R2 and MinIO too.
  • CSV / log upload: drop a Cloudflare CSV export, an AWS CloudFront access log, an Apache or nginx log, or any CSV with timestamps and user agents. Gzipped files work too.
• Enterprise accounts also get a Per URL tab showing exactly which pages each AI bot crawled, with hits, status codes, and first/last-seen dates, plus CSV export

How to use it

1. Open Bots & Traffic > Agent Analytics
2. Connect a data source (Cloudflare is the fastest if your site is behind it)
3. Give the first sync a little time, then review which AI bots visit you and how often
4. Watch the trend after publishing new content: rising AI crawler traffic on your site is an early signal that models are picking it up

Connect an Amazon S3 bucket

If your server or CDN already ships access logs to an S3 bucket, you can let LLM Pulse read them directly instead of uploading files by hand.

What it is: a read-only connection to a bucket of log files. Once a day (plus a manual "Sync now") we list the bucket, pick up only the log files added since the last sync, parse them, and match each request against the AI bot catalog. It also works with S3-compatible stores such as Cloudflare R2 and MinIO through an optional custom endpoint URL.

Before you start: create an IAM user with a read-only policy that allows s3:GetObject and s3:ListBucket on the bucket (and prefix, if you use one), then generate an access key ID and secret access key for it. A minimal policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::your-log-bucket",
        "arn:aws:s3:::your-log-bucket/*"
      ]
    }
  ]
}

Set it up:

1. Open Bots & Traffic > Agent Analytics
2. On the Amazon S3 card, enter your bucket name, AWS region, and (optionally) a key prefix so we only read the folder that holds your logs
3. Paste the read-only access key ID and secret access key. For Cloudflare R2, MinIO, or another S3-compatible store, also add the custom endpoint URL
4. Click Connect. Your credentials are stored encrypted, and the first sync starts automatically. Use Sync now any time you want to pull the latest files without waiting for the daily run

Sync cadence and limits: we sync once a day and on demand via "Sync now", reading only files that are new since the last sync. We ingest log files dated up to 90 days back, up to 256 MB per file, and up to 500 new files per sync.

Supported log formats: Apache and nginx combined (CLF) logs, AWS CloudFront standard (W3C) logs, and CSV log exports. Gzipped (.gz) files are read directly.

Tips & notes

• Agent Analytics is the inverse of AI Traffic: AI Traffic measures humans arriving from AI tools, Agent Analytics measures the AI crawlers themselves visiting your site
• Verified bot detection combines the provider's bot classification with our own user-agent classifier, so spoofed user agents are not over-counted
• If your CDN or host is not listed, the CSV/log upload path accepts most standard log formats
• Per-URL crawl data requires an Enterprise plan and a connected Cloudflare source