Google Analytics never sees most AI bots. The script that logs a visit only fires inside a browser, and most crawlers never open one.
I check a different record instead: the raw server log. It writes down every request before any script has a chance to run, so it catches what analytics was built to miss.
What a Server Log Records
Every request to a server leaves a line. The path asked for, the time, the response code, and the user agent string naming whatever made the request.
A person's browser writes one of these lines. So does a training bot. So does a script scraping a pricing page at three in the morning. The log does not care who is asking. It writes down that someone did, and moves on.
That is both the appeal and the limit of log data. It is complete. It is also unlabeled, and reading it takes work analytics was built to skip entirely.
Why Analytics Misses the Crawl
I wrote before about what AI referral traffic looks like inside analytics. That post covers clicks, a person reading an answer and following a link back to the site. This is a different layer underneath it.
Most bots that feed an AI engine's index never click anything. They fetch a page, read the text, and move on, no browser involved and no script fired. Analytics counts visitors. It was never built to count readers that leave no session behind.
The log is where that visit actually lands. If GPTBot or ClaudeBot has touched the site even once, it shows up there and nowhere a dashboard would show it.
The Bots Worth Watching For
A short list covers most of what matters. GPTBot and OAI-SearchBot for OpenAI. ClaudeBot for Anthropic. PerplexityBot for Perplexity. Google-Extended for Gemini training, which is separate from the Googlebot that handles search and AI Overviews. Applebot-Extended for Apple. Bingbot, feeding several assistants downstream.
I filter the raw access log for each name directly. A clean match on the user agent field is usually enough on its own.
- A site with real content and an open robots.txt should show hits from several of these in the log.
A site that shows none, month after month, has a different problem than a content problem.
One Hit Proves Almost Nothing
A single crawl on one day does not mean an engine is reading a site seriously. Bots sample constantly, and most of what they touch never gets indexed or cited anywhere.
What actually matters is the pattern over time. Which pages get hit again and again. Whether a new post gets crawled soon after going live, or sits untouched instead. Whether the attention lands on the pages built to answer real buyer questions, not the thin ones nobody should be linking to.
I watch frequency on the pages meant to carry weight: the service pages, the FAQ sections, the pages carrying real structured markup. Repeated attention there means more than a single visit anywhere else.
When a Blocked Bot Shows Up Anyway
robots.txt is a request, not a lock. Most major AI bots respect it. Not every one does, and the log is the only place that would actually tell you.
If a bot named in the disallow list keeps showing a 200 response instead of being turned away, the block is not working the way the file claims. Worth checking against the live robots.txt, not the one assumed to be deployed.
The reverse case costs more. A bot meant to be welcomed should log a 200, not a 403. I have found sites blocking one of them by accident through a CDN rule or a bot-management setting nobody ever connected to AI visibility.
What the Pattern Is Actually Worth
None of this proves a ranking, a citation, or a share of answers inside any engine. It proves access. A bot that cannot read a site cannot recommend it, and the log is the one record that confirms the bot got in at all.
Checked once, it is a diagnostic. Checked monthly, alongside robots.txt and the structured markup on the pages that carry it, it is maintenance. Few businesses do either, which is exactly why the ones who do notice a problem months before anyone else would.
This kind of access check runs inside the SEO / AEO / GEO Foundation, built alongside the access and structure work it depends on. Structure only. No ranking or citation guarantees.