Three different things called “AI traffic”
Almost every confused conversation about AI and analytics is two or three questions wearing one label. Separating them makes each one answerable:
- Crawling. A bot fetching your pages to train or ground a model. Costs you bandwidth and server capacity. Measured in server logs.
- Referral. A person clicking a link inside an AI answer. Real traffic, real browser, arrives with a referrer. Measured in your analytics.
- Agentic browsing. An AI agent operating a real browser on someone’s behalf. Executes your tag. Measured in your analytics, whether you want it there or not.
They have almost nothing in common operationally, and the first one — the one that generates the headlines — is the one your analytics cannot see at all.
The number everyone quotes, and what it does not mean
Imperva’s 2026 Bad Bot Report, drawn from 17.2 trillion blocked bot requests across its network, put automated traffic above 53% of all web traffic during 2025, up from 51% the year before, split roughly into 13% “good” bots (search crawlers, AI crawlers, monitoring) and 37% bad ones. Human activity is now the minority of requests on the web.
What that figure describes is requests, not the contents of your analytics dashboard. Those two populations barely overlap, and the reason is worth understanding precisely, because it determines where you should be looking.
AI crawlers do not run your JavaScript
A client-side analytics tag is JavaScript. It produces an event only when a browser downloads it, parses it and executes it. Anything that fetches raw HTML and reads the markup — which is what most crawlers do — never triggers it, and never appears in your reports.
This is measured, not assumed. A joint analysis by Vercel and MERJ examined more than 500 million GPTBot fetches and found no evidence of JavaScript execution at all. GPTBot, ClaudeBot and PerplexityBot fetch HTML and extract text from the initial markup. They do request JavaScript files — roughly a tenth of ChatGPT’s requests and closer to a quarter of Claude’s — but downloading a script is not running it.
Two exceptions matter. Gemini rides on Google’s crawling infrastructure and therefore renders pages the way Googlebot does. AppleBot also renders in a real browser. Both can, in principle, execute a tag — Googlebot’s rendering pass is on every serious crawler list for exactly this reason.
If your analytics is clean of AI crawlers, that is not your bot filtering working. It is JavaScript doing the filtering for you, and it filters indiscriminately.
So where does the load actually show up?
In the places that count requests rather than page loads: your server logs, your CDN dashboard, your bandwidth bill, your cache hit ratio, your origin CPU. This is why the “my server is at 90% and analytics shows nothing” pattern is so common — the two systems are not in conflict, they are measuring different populations. Analytics measures browsers that ran your script. Infrastructure measures every request that arrived.
The practical consequence: if you want to know what AI crawlers cost you, do not go looking in your product analytics. Go to the logs. And if you do see a named AI crawler in your analytics reports, one of three things is true — your tool measures server-side, something is spoofing that user agent, or you are looking at referral traffic and mistaking it for crawling.
The crawl-to-referral question
The reason anyone tolerates crawler load is the hope of citations, and it is fair to ask what the trade returns. Third-party trackers publish a crawl-to-referral ratio: how many pages a crawler took for each visit its platform sent back. The ratios are large — hundreds of pages crawled per referred visit for the best performers, thousands for the worst — and they have been improving over the past year as AI products send more traffic out.
Treat any specific figure as directional. It is computed from different networks with different audiences and it moves month to month. The shape is the useful part: crawling is expensive and referral is small but growing, so the honest framing is an investment with a lengthening payback, not a fair exchange happening today.
Measuring your own side of it is straightforward, and worth doing rather than reading someone else’s ratio. AI referrals arrive as ordinary traffic with an identifiable referrer, so a breakdown by referring domain, filtered to the assistant domains, gives you a number specific to your site.
The category that will actually reach your tag
Agentic browsing is the one to watch. An agent that drives a real browser — booking something, comparing prices, filling a form on a person’s behalf — executes your JavaScript, produces pageviews, and moves through your funnel. Some of it is arguably a person’s traffic, since a person asked for it. Some of it is pure automation, and it will distort your metrics the same way any bot does.
This is hard to classify honestly. A browser under automation looks like a browser. The signals that catch it are a declared user agent, an automation flag the browser exposes, or an origin network that only hosts servers — and an agent that avoids all three is not catchable by any free signal. Anyone claiming to detect agentic traffic reliably at the analytics layer is describing an ambition.
What to do
- Ask which question you are answering. Crawler cost lives in logs, referral lives in analytics, agent traffic lives in your bot classification. One dashboard will not do all three.
- Do not “fix” your analytics for crawlers that were never in it. Adding rules for GPTBot to a client-side tool is effort spent on traffic that cannot reach it.
- Watch the gap. A widening divergence between requests at your edge and pageviews in analytics is the crawler trend showing up in the only place it can.
- Segment AI referrals explicitly. They are small enough that they disappear into Direct or Other unless you name them, and they are the number that tells you whether being crawled is paying off.
- Keep robots.txt a separate decision. Blocking crawlers is a licensing and visibility choice. It will not clean your analytics, because your analytics was already clean of them.
How Pug treats it
Pug measures the browser, so the same physics applies: a crawler that does not execute the SDK produces nothing. What it adds is that the rare case is handled honestly — when something on the crawler list does run the web SDK, a rendering pass or an agent that declares itself, the event is tagged with the matched name rather than silently dropped, and excluded from metrics by default. The reason is stored as a queryable dimension, so “which machines reached us, and how many” is answerable instead of assumed.
Everything above the tag — the crawl volume, the bandwidth, the cache pressure — is a logs and CDN question, and no client-side analytics tool can answer it. Any that claims to is measuring something else.