Agent Analytics: AI Bot Crawler Tracking

Before an AI model can cite your site, its crawlers have to visit it. Agent Analytics is server-side AI bot traffic monitoring: it shows which AI crawlers (GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot, Google-Extended, and many more) are hitting your website and how that evolves over time.

With Agent Analytics, you can:

See which AI bots (GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot, Google-Extended) crawl your site and how often
Confirm that AI models are actually discovering new or updated pages before you expect them in AI answers
Compare crawl volume by company (OpenAI, Anthropic, Google, Perplexity, and more) to see who is investing in your content
Catch a drop in AI crawler activity that could explain falling citations

Find it under Bots & Traffic > Agent Analytics in the sidebar. Available on Scale plans and above.

In embed and full-whitelabel portals, the client connects and manages their own sources: Cloudflare, Bunny CDN, an Amazon S3 or CloudFront log bucket, and CSV or log uploads. Log drains are also available when the portal is served on the client's own domain, because a drain hands out an ingest URL and that URL has to sit on a host the client recognises. Where drains are not offered, a portal cannot disconnect an existing log drain or connect Cloudflare, Bunny CDN or an Amazon S3 or CloudFront log bucket in its place, because that would delete the drain's history. Asking us to support a provider we do not cover stays in the main workspace. Team member permissions apply as everywhere else, so a portal user who cannot create connections still sees the report and is told to contact their account manager.

To change what a saved source reads (another Cloudflare zone, another Bunny pull zone, or a different Amazon S3 region, prefix or endpoint), the portal enters its own token, key or S3 key pair again, and can only pick a zone that token or key reaches. Saving an S3 source without changes, for example to resume it after a pause, does not need the keys.

What it does

Tracks over 40 AI bots grouped by company (OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Microsoft, Amazon, and others)
Daily traffic charts per bot and per company, so you can see whether AI crawlers are discovering your content
Multiple ways to get your data in:
- Cloudflare integration: paste a scoped API token (Zone Analytics: Read), pick a zone, and data syncs daily. Works on every Cloudflare plan, including Free.
- Bunny CDN integration: paste your account API key and pick a pull zone; data syncs daily.
- Log drains (real time): Vercel, Akamai DataStream 2, Fastly, Netlify, and GCP Load Balancer. Generate a unique ingest URL in the app and paste it into your provider's log drain settings.
- AWS CloudFront / Amazon S3 bucket: connect the S3 bucket where CloudFront or your server writes access logs and we pull new files once a day (plus a manual "Sync now"). CloudFront standard W3C logs work directly. S3-compatible stores such as Cloudflare R2 and MinIO work too.
- CSV / log upload: drop a Cloudflare CSV export, an AWS CloudFront access log, an Apache or nginx log, or any CSV with timestamps and user agents. Gzipped files work too.
A Per URL tab showing exactly which pages each AI bot crawled, with hits, status codes, and first/last-seen dates, plus CSV export

How to use it

Open Bots & Traffic > Agent Analytics
Connect a data source (Cloudflare is the fastest if your site is behind it)
Give the first sync a little time, then review which AI bots visit you and how often
Watch the trend after publishing new content: rising AI crawler traffic on your site is an early signal that models are picking it up

Per URL analysis

The Per URL tab lists the pages each AI bot requested, so you can see which content GPTBot, ClaudeBot or PerplexityBot actually reads. It works with every source: Cloudflare, Bunny CDN, log drains, Amazon S3 or CloudFront buckets, and log uploads. Query strings are ignored and static files such as scripts, styles, images and fonts are left out, so the table shows your pages. Cloudflare brings in up to the last 30 days the first time Per URL syncs (when you connect it, or after you move to Scale or a higher plan) and later fills in any days a failed sync missed, while Bunny CDN only keeps the last 3 days. Log drains start with the next delivery, the first sync after you connect an Amazon S3 or CloudFront bucket reads the log files already in it (entries up to 90 days old), and uploads cover the days in the file.

Uploads leave out lines dated after tomorrow or more than 395 days ago. Per URL takes up to 500,000 rows from each file (one row is one URL, one bot and one day).

Per URL analysis uses monthly credits, shared by all the projects in your account:

One credit is one URL, one bot and one day. More visits from the same bot to the same URL on the same day cost nothing extra.
Scale includes 100,000 credits a month, Scale+ 200,000 and Scale++ 400,000. Higher plans include more.
Credits renew on the 1st of each month and do not roll over. The tab shows how many you have used, how many this project used and when they renew.
When credits run out, the URLs, bots and days already tracked keep updating, and new ones wait until the 1st. The Overview keeps counting every bot visit. You get an email at 80% and at 100%, and nothing is ever charged automatically.
Need more? Add a recurring credit pack (+100,000, +500,000 or +1,000,000 a month) from Billing & Plans.

Under Bots analyzed per URL you choose which bots each project analyzes per URL. Every bot is on by default, and each one shows the credits it used this month. A bot you turn off stops using credits from the next sync. Its history stays, the Overview still counts it, and it is not blocked on your website.

Per URL detail is kept for 90 days.

Set up Vercel

In Vercel, go to Team Settings > Drains > Add Drain and choose Logs as the data type. Select the projects you want to track and include Production. Enable the Static, Lambda, Edge, External, Firewall, and Redirect log sources. Leave Build off because it contains deployment output, not visitor traffic. Do not add sampling rules, so Vercel forwards 100% of requests.

Use Custom Endpoint with NDJSON and paste the LLM Pulse endpoint URL. Web Analytics, Speed Insights, and Traces are not supported because they do not include the request user agent needed to identify AI bots. The Signature Verification Secret is optional in LLM Pulse. Copy Vercel's generated value into Signing secret only if you want to verify signed requests. Enter a verification code in LLM Pulse only if Vercel shows one.

Connect AWS CloudFront or another Amazon S3 log bucket

If CloudFront, your server, or another CDN already sends access logs to an S3 bucket, you can let LLM Pulse read them directly instead of uploading files by hand.

What it is: a read-only connection to a bucket of log files. Once a day (plus a manual "Sync now") we list the bucket, pick up only the log files added since the last sync, parse them, and match each request against the AI bot catalog. It also works with S3-compatible stores such as Cloudflare R2 and MinIO through an optional custom endpoint URL.

CloudFront format: legacy CloudFront standard logs sent to S3 already use the supported W3C, tab-separated, gzip-compressed format. For standard logging v2, choose Amazon S3 as the destination, W3C as the output format, and Tab as the field delimiter. Keep date, time, cs(User-Agent), cs-uri-stem, and sc-status. We also recommend sc-bytes for bandwidth reporting. JSON, Plain, Raw, and Parquet output are not supported.

Before you start: create an IAM user with a read-only policy that allows s3:GetObject and s3:ListBucket on the bucket (and prefix, if you use one), then generate an access key ID and secret access key for it. A minimal policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::your-log-bucket",
        "arn:aws:s3:::your-log-bucket/*"
      ]
    }
  ]
}

Set it up:

Open Bots & Traffic > Agent Analytics
On the AWS CloudFront card, click Connect via Amazon S3. For another S3 log source, use the Amazon S3 card. Enter your bucket name, AWS region, and (optionally) a key prefix so we only read the folder that holds your logs
Paste the read-only access key ID and secret access key. For Cloudflare R2, MinIO, or another S3-compatible store, also add the custom endpoint URL
Click Connect. Your credentials are stored encrypted, and the first sync starts automatically. Use Sync now any time you want to pull the latest files without waiting for the daily run

Sync cadence and limits: we sync once a day and on demand via "Sync now", reading only files that are new since the last sync. We ingest log entries up to 90 days old from files up to 256 MB each. We process the files in batches of up to 500 and start another batch automatically until the bucket catches up. We scan up to 50,000 objects in the configured bucket or prefix; use a narrower prefix if it contains more.

Supported log formats: Apache and nginx combined (CLF) logs, AWS CloudFront standard (W3C) logs, and CSV log exports. Gzipped (.gz) files are read directly.

Tips & notes

Agent Analytics is the inverse of AI Traffic: AI Traffic measures humans arriving from AI tools, Agent Analytics measures the AI crawlers themselves visiting your site
Verified bot detection combines the provider's bot classification with our own user-agent classifier, so spoofed user agents are not over-counted
If your CDN or host is not listed, the CSV/log upload path accepts most standard log formats
Per URL data is available from Scale with any connected source, within your monthly credits