TL;DR
- More AI bot requests show more recorded activity. They do not, on their own, show more citations, buyers or revenue.
- Training crawls, search crawls and user-triggered page visits serve different jobs. Sort them by purpose first.
- Research on crawl volume and purpose helps explain the gap, but network-wide numbers are not a benchmark for your site.
- Use logs to find access failures. Use saved answers to check citations and separate analytics to check human visits.
Separate a request from an answer
Ask what changed and what the log can help fix. A blocked search crawler and a busy training crawler can produce very different work priorities, even when both appear under an “AI traffic” heading.
Why One AI Traffic Total Can Mislead
Cloudflare's August 2025 analysis classified 80% of AI crawling as training, 18% as search and 2% as user actions over its reported twelve-month period. Cloudflare used bot owners' statements and industry sources to group the requests. These are past network results. They do not show the current mix on your site.
| Classified purpose | Share of AI crawling |
|---|---|
| Training | 80% |
| Search | 18% |
| User actions | 2% |
Source: Cloudflare, August 29, 2025, “Crawling by purpose.” Twelve-month view reported in the article; not the separate six-month breakdown.
The mix explains why more requests need not mean more buyers. Requests grouped under one label can serve tasks far removed from a buyer's search. The chart also does not mean that search crawls turn into clicks at a fixed rate. Each stage needs its own evidence.
Historical network data classified using operator disclosures and industry sources. This is not the current mix on your site or the separate six-month breakdown. Cloudflare, August 2025.
Big Crawl Counts Measure Activity, Not Results
In Vercel and MERJ's December 2024 study, GPTBot made 569 million requests across Vercel's network in the prior month. The report also recorded 370 million for Claude and 24.4 million for PerplexityBot. Those volumes describe requests, not unique buyers or citations. The study tracked Vercel's network and a few sites. It is old data. It cannot set current rules for how every bot works.
| Direct research | Observation | Actual result | Limit for a B2B report |
|---|---|---|---|
| Cloudflare, August 2025 | AI crawl activity grouped by purpose | Training 80%, search 18%, user actions 2% | Purpose shares do not establish answer use or referral rates |
| Vercel and MERJ, December 2024 | Requests across Vercel's network in the preceding month | GPTBot 569M; Claude 370M; PerplexityBot 24.4M | Fetch counts do not identify unique people or useful citations |
The studies add two views. One shows why the traffic exists. The other shows how large request counts can be. Neither tells you how many crawls turn into cited answers. Do not combine their figures into a ranking of which platform sends the best leads.
Current bot roles also matter. OpenAI's crawler documentation distinguishes OAI-SearchBot for search, GPTBot for content that may be used in model training, and ChatGPT-User for certain user-triggered actions. Search and training controls are separate, and OpenAI notes that a crawl may serve both uses when both are allowed. So a bot label alone does not prove how the page was used.
Turn the Log into a Useful Decision
Before you explain a spike, check who sent the request. Then check the URL, status and time window. Use the bot owner's current guide to check its identity. A name in a user-agent string can be faked. Keep retries, errors and repeated fetches visible rather than counting each as a new opportunity.
Use this table to choose the next check. These are our suggestions. They are not results from the network studies.
| Observed signal | What it can support | What it cannot establish | Next check |
|---|---|---|---|
| Verified search bot receives an access error | A possible access problem for that request | That the page would otherwise be cited | Inspect the error and the affected content with the site owner |
| Training bot requests increase | More recorded fetch activity by that bot | More search appearances or buyers | Review purpose and operational cost separately |
| User-triggered fetch receives a full page | The tool could access the response in that request | A human click, a recommendation or a sale | Inspect the saved answer if available |
| A public answer links to the page | A citation in that recorded answer | Reliable support for every nearby claim | Check the cited passage and preserve the prompt and date |
Worked example — fictional engineering supplier: a report calls a rise in GPTBot requests an “AI visibility gain.” The analyst separates bot types and finds no saved buyer answers in the report. She changes the claim to “training crawler activity increased.” In a separate check, a search bot received an error on a capability page. The site owner investigates that error. The team has found a real issue. It has not claimed a gain it cannot show.
Our AI search citation diagnostic covers the wider source check. Logs show whether a request got through. Keep the saved answer as well. The buyer's question tells you whether that answer matters.
AI Bot Traffic: Common Questions
Should all AI bots be allowed?
That is a choice for the site owner. It depends on what the bot does and what your business needs. A crawl report alone cannot settle it. Review current operator documentation with the site owner before changing access rules, and keep search access separate from training choices.
Does a successful response prove that the bot read the whole page?
No. A successful status code shows that a response was served. Inspect what the response contained and, where possible, what the tool actually used. A fetched shell with little text is different from a full product specification.
Can we divide referrals by crawls to get an AI conversion rate?
Not a buyer conversion rate. Those counts may involve different bots, purposes, pages and people. A crawl-to-referral ratio can describe a defined traffic relationship, but it does not track one person from a crawl to a sale.
What is a sound success claim after fixing an access issue?
State what you checked: the right bot could fetch the right page in your test. Check later citations and human visits separately. Do not turn an access fix into an unmeasured claim of greater AI reach.
Review the signals behind the dashboard
If a bot-traffic dashboard is driving content decisions without clear buyer-answer evidence, bring the relevant page URLs and a redacted log summary to SemShift's AI visibility audit. We can discuss which access issues and source gaps deserve a focused review.