Which pages are AI bots crawling?

Is AI crawling my site? tells you whether a simulated request from Knowatoa can reach your homepage. This article answers the next two questions:

Which of my pages are crawlers reading for an index or a training set?
Which of my pages are agents fetching live, on behalf of a person in ChatGPT, Claude, or Perplexity?

Those answers live in the logs of the service that actually sits in front of your site. Cloudflare and Microsoft Clarity can show them without turning off Bot Fight Mode.

Crawlers and agents are not the same request

A crawler walks your site on a schedule. It stores what it finds for search or for model training. In the AI Search Console these are names like GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot.

An agent fetches a page in response to a person. It needs the current HTML, not last week’s snapshot. Those User-Agents are ChatGPT-User, Claude Web, and Perplexity-User.

A page that crawlers visit every night can still fail when an agent asks for it in a chat. The reverse is also true. Filter the logs by bot name. Do not treat “AI traffic” as one number.

A request in these logs means the bot asked for the URL. It does not mean the model cited you, grounded an answer on that page, or sent you a visitor.

First check: the AI Search Console

Keep using the AI Search Console. It is the fast, first-line check: can a request that claims to be GPTBot or ClaudeBot reach the site from our IPs?

It cannot list the pages the real bots read. For that, open the service that already sees their verified traffic.

Cloudflare: AI Crawl Control

If the site is on Cloudflare (orange-cloud proxy), you already have the data. Bot Fight Mode can stay on.

Open the Cloudflare dashboard and select the domain.
Go to AI Crawl Control.
On Overview, confirm that real operators (OpenAI, Anthropic, Google, Perplexity) appear, and whether their requests were allowed.
Open Metrics.
Use the crawler or operator filter. Switch from GPTBot to ClaudeBot to OAI-SearchBot (crawlers), then to ChatGPT-User or the equivalent agent name (live retrieval).
Open Most popular paths. That table is the page list: path, hostname, and how many of those requests Cloudflare allowed.

What to look for

Allowed 2xx rows on the pages you care about — the real bot is reading those URLs.
Volume on / and /robots.txt only — the bot reached the site but is not walking the rest of it.
A 4xx stack for one crawler — a rule or AI-crawler block is stopping that bot. That is different from our simulated test failing. See Bot Failures & Investigations.

Limits

The table is top paths, not every URL the bot requested.
On the Free plan, Metrics covers the last 24 hours.
Security → Events shows blocked or challenged hits, including Knowatoa’s spoofed test. It is not a map of pages the real bots successfully read.

Cloudflare groups some traffic as Search, Agent, or Training. Use that grouping when you need “index crawl” versus “live fetch” without memorizing User-Agent strings.

Microsoft Clarity: Bot Activity

Clarity’s session recordings do not see GPTBot. Those crawlers do not run JavaScript. Use AI Visibility → Bot Activity, which reads server-side CDN logs, not the Clarity snippet.

Cloudflare is a supported source. If the site is already on Cloudflare, connect that zone. You do not need to turn off Bot Fight Mode.

Sign in to Microsoft Clarity and open the project.
Go to Settings → AI Visibility and connect a supported CDN (Cloudflare, CloudFront, Fastly, Azure Front Door, or Akamai).
After logs start flowing, open Dashboards → AI Visibility → Bot Activity.
Use the operator and bot breakdown to pick one crawler or one agent.
Open Path requests. That list is the pages and assets those bots requested.

Clarity can group paths by page type (blog, product, and so on). Use that when you care about a section of the site more than a single URL.

Connecting a CDN may add cost on the CDN side. Clarity bills that through the provider, not as a Knowatoa fee.

If you have neither Cloudflare nor Clarity

Ask whoever hosts the site for access logs, filtered to the vendors’ published crawler IP ranges. The question is the same: which paths did GPTBot hit, and which paths did ChatGPT-User hit?

The end-to-end product check is still a branded question in Knowatoa. If the model cites your pages, the crawl and the retrieval both worked. See Persistent Console Failures.