What counts as a bot
The category is broader than “crawlers”, and the members behave nothing alike:
- Search and AI crawlers — Googlebot, GPTBot, ClaudeBot. They fetch HTML to index or ground a model, and mostly do not run JavaScript.
- Uptime and synthetic monitors — a real browser loading one page every minute, forever, from a datacenter.
- Headless browsers — Playwright, Puppeteer and Selenium, driving a full Chrome with no screen. Your own end-to-end tests are in this group.
- Performance tooling — Lighthouse and page-speed checks, which load and render the page exactly as a person would.
- Scrapers — some parse raw HTML, some render pages because that is the only way to read a JavaScript-heavy site.
- Agentic browsing — an AI agent operating an ordinary browser on someone’s behalf, which is the hardest of all to classify.
The distinction that matters for analytics
Not whether a bot is “good” or “bad”, but whether it executes JavaScript. A client-side analytics tag only produces an event when a browser runs it, so a crawler fetching raw markup is invisible to it no matter how many pages it takes. That is why AI crawlers barely appear in analytics reports while dominating server logs — and why the automation that does pollute your reports is mostly monitors, headless browsers and rendering scrapers.
The bots in your server logs and the bots in your analytics are largely two different populations. Filtering for one does very little about the other.
How bot traffic is detected
Three signals are available without a commercial bot-management service, and each has a specific blind spot:
- User-agent lists. Match the request’s user agent against a maintained list of crawler, monitor and automation patterns. Cheap and precise, but it only catches automation that identifies itself.
- Origin network. Check whether the request came from a network that only hosts servers — AWS, Hetzner, DigitalOcean and so on. Catches automation that hides its identity, at the cost of flagging real people behind cloud-hosted VPN exits and corporate proxies.
- Browser automation flags. A browser under automated control exposes signals a script can read. Useful as an optimisation, but the browser is under the bot’s control, so anything it reports can be suppressed.
What none of them catch is automation running a stock browser from a residential or mobile address. Bot networks doing exactly this exist, and no free signal identifies them.
Why it distorts metrics in one direction
Bot traffic is not random noise that averages out. It has a consistent shape — one page, no referrer, no engagement, perfectly regular timing — so it biases every metric the same way each day: inflated sessions, a permanent floor under bounce rate, a deflated conversion rate, and a retention curve propped up by monitors that return every single day. Which metrics break, and which way covers this in detail.
Tagging versus dropping
Tools handle detected bots in one of two ways. Most drop: the traffic is discarded before storage, so it produces no count and leaves no record. Some tag: the event is stored with a marker and left out of the numbers when you run a report — so how much was excluded stays measurable, and the decision stays reversible.
The difference only matters when detection is wrong — which it will be, because every signal above has false positives. A dropped hit from a real visitor behind a corporate proxy is permanent, silent data loss. A tagged one is a filter you can switch off. Analytics tools differ sharply here.
How Pug handles it
Pug matches the incoming user agent against a maintained list of around 1,500 crawler and automation patterns and
tags the event rather than dropping it, recording both that it was automated and which pattern
matched — HeadlessChrome, Googlebot, a named synthetic-monitoring service. Both
properties are set on the server, so nothing running in the visitor’s browser can add or remove them.
Every metric excludes tagged traffic by default, and session metrics judge a whole session rather than individual hits, so a visit with any automated activity is dropped entire instead of leaving a truncated fragment behind. On the raw event surfaces a single control brings the traffic back into view, and the recorded reason is available as a breakdown, so how much was removed is a query rather than a guess.