We logged 101,024 AI crawler hits on our own site. Here is who reads what

The module reads the web server's request logs and identifies AI crawlers by their user-agent names. Each request is one hit. Three limits matter before any number below.

• A hit is a fetch, not a reader. One person's question to an assistant can trigger several fetches. Hits tell you that an engine requested a page. They do not tell you that anyone read it, or that the page was cited.
• User-agent names can be faked. The module checks requests that claim to be AI crawlers and excludes those that fail. On September 22 it had excluded 9,604 requests as spoofed. The numbers below are the ones that passed.
• Categories are our module's labels. It sorts bots into training, user request, search index and a few smaller groups. Those labels come from our classifier and from how each vendor describes its bots, so read them as a working map.

Two readings of the same module appear in this article, and we keep them apart. The September 22 snapshot (101,024 hits, 24 bots, observation from June 23) comes from the module's dashboard. The October 13 reading (123,378 hits, 26 bots, same start date) comes from a fresh query of the same module for the per-bot and per-page tables, which the dashboard screenshot does not show. The ranking of the top bots is the same in both.

The September 22 split by bot type:

More than half of all hits come from bots that collect content for model training. A quarter come from bots that fetch a page because a user asked an assistant something. The smallest of the three large groups, about one hit in six, is search indexing.

The October 13 reading, with the 15 busiest bots out of 26:

Three bots account for 57% of all hits: meta-externalagent, ChatGPT-User and Amazonbot. The top 15 together cover 98.6%.

Grouping the same table by company shows how differently the engines use a site.

• OpenAI sends three bots. ChatGPT-User made 20,831 requests, GPTBot 5,348 and OAI-SearchBot 4,532. The user-triggered fetcher is about 4.6 times busier than the search crawler. OpenAI documents these as separate bots with separate jobs, and says each robots.txt setting is independent of the others.
• Perplexity is mostly an indexer: PerplexityBot 8,695 against Perplexity-User 4,113.
• Anthropic is mostly a collector: ClaudeBot 10,372 against Claude-User 2,126.
• Meta appears twice, as meta-externalagent (36,683, the single busiest bot) and as meta-webindexer (1,711).

One thing the log does not explain: the busiest bot on our site, meta-externalagent, belongs to a company that does not appear among our AI referral sources at all. Crawl volume and referral volume are separate quantities, and a high number in one tells you nothing about the other.

The module also ranks pages by hits and names the busiest bot on each. The top 15 paths on October 13, out of 1,113 pages requested:

Five things stand out.

1. Bots read the rules before the content. The most requested path is robots.txt, with 5,336 hits, or 4.3% of all traffic. The sitemap index adds 1,399. Crawlers re-read these files constantly, which is why a mistake in robots.txt is expensive: it is the first file every bot looks at. Our knowledge base calls this the first gate, Fetchable: if a page cannot be fetched, nothing else about it matters.

2. On every blog article in the top 15, ChatGPT-User is the busiest bot. Seven articles are on the list, and the user-triggered fetcher leads on all of them. That fits a simple reading: when a person asks ChatGPT a question and the assistant decides to open a page, the page it opens is one of these. Our reading is a hypothesis, because the log does not record the question.

3. The pages that attract fetches are not only the new ones. Articles about content marketing statistics, tripwire funnels and Facebook ads agency pricing were written for an earlier version of the blog, and they rank among the most-fetched pages next to the AEO and GEO comparisons we wrote on purpose. A fetch means the assistant found the page relevant to a question. It says nothing about whether the page was cited, or whether it should be.

4. Product and service pages get crawled too. /platform, the homepage and a services page together take about 13,000 hits, from a mix of indexing, training and user-triggered bots. These are the pages that explain who we are, and they are being read by the machines that will describe us.

5. Some hits are housekeeping. The favicon alone got 3,031 hits. A log of raw requests includes files nobody would call content, so we read page-level numbers as directional.

Yandex Metrica, read through the same workspace for the 365 days from October 13, 2025, counted 119 visits from AI sources out of 5,112 total, or 2.3%. By source: OpenAI 79, Perplexity 15, Anthropic 10, Gemini 10, Alice 2, DeepSeek 2, Copilot 1.

Set against the crawler log, the proportions are lopsided. Over the same site, there are roughly a thousand crawler hits for each referred visit (123,378 against 119; the log starts June 23 and the visits cover a full year, so the real ratio is, if anything, higher). ChatGPT-User fetched pages 20,831 times, while OpenAI referral visits numbered 79.

We do not read that as waste. A fetch is the assistant gathering material for an answer, and most answers produce no click. This is the same gap we describe in Zero-click: what GA4 shows about AI traffic and what it misses: the click is the smallest visible part of what happens.

On September 22 the module had excluded 9,604 requests that claimed to be AI crawlers and failed verification. By October 13 the excluded count was 12,664. Against the 123,378 accepted hits, that is roughly 9% of all requests claiming an AI user agent.

A user-agent string costs nothing to copy, so any log that counts bots by name alone will overstate AI crawling. If you read raw logs, check claimed crawlers against the IP ranges the vendors publish. OpenAI, for one, publishes IP lists for its bots.

Six practical steps, ordered from cheapest to most work.

1. Check robots.txt and your CDN first. If a crawler is blocked by a firewall rule, it will never reach your content. OpenAI states that its search bot needs access to your site, and that its settings for search and training are independent.
2. Decide per bot type, not per company. Training, search indexing and user-triggered fetching are different jobs. Allowing OAI-SearchBot while disallowing GPTBot is a valid choice, and so is any other combination. Make it on purpose.
3. Count user-triggered fetches separately. Hits from ChatGPT-User, Perplexity-User and Claude-User are the closest thing a log offers to "an assistant opened this page for a person".
4. Filter spoofs before you report. Report accepted hits, and mention the exclusion rate.
5. Look at pages, not totals. A rise in total hits can come from one page being crawled in a loop. Group by path and by bot.
6. Put logs next to analytics. Logs show fetches, GA4 and Metrica show clicks, and a visibility tool shows who gets named. Each answers one question.

• It does not say which fetched pages were cited in an answer. That needs a visibility tool that asks the engines your prompts.
• It covers one B2B marketing site. A shop, a news site or a developer-docs site would see a different mix.
• The snapshot and the October 13 reading cover different end dates, and we do not mix them in any calculation except where we say so.
• Bot categories are our module's labels, not the vendors'.
• We make a product in this category, and this module is part of it. Every number here comes from our own logs, including the ones that make our site look ordinary.

If you want to compare tools that measure the visibility layer, our market comparison is here: AI visibility tools in 2026: 10 platforms compared with pricing.

• Humanswith.AI workspace, AI-traffic module (bot traffic view), dashboard snapshot of September 22, 2026 and live reading of October 13, 2026, own data
• Yandex Metrica, AI referral visits since October 13, 2025, read through the Humanswith.AI workspace, own data
• OpenAI, Overview of OpenAI crawlers
• Humanswith.AI, Zero-click: what GA4 shows about AI traffic and what it misses
• Humanswith.AI, AI visibility tools in 2026: 10 platforms compared with pricing