TL;DR
- Treat llms.txt as an optional resource for a defined tool or agent, rather than a default AI search project.
- Google says these files do not help or harm visibility in Google Search, including its generative AI features.
- Request logs and citation studies measure different outcomes. Neither a file's presence nor a bot fetch proves that an answer used it.
- Fix access to useful pages and gaps in buyer evidence first. Give any file experiment a clear owner and success check.
Ask what the file is meant to change
Before paying for a file or treating its absence as an urgent audit failure, ask what it is meant to change. A document can be valid, get fetched and still have no demonstrated effect on the outcome you care about.
What is the file supposed to do?
The llms.txt proposal, updated in August 2026, describes a Markdown guide that gives agents context and links to useful material. It can cover a whole site or a section such as software docs. It serves a different purpose from robots.txt, which communicates access rules to automated tools.
That distinction matters for a B2B site. A guide to your service pages cannot fix a broken service page. It cannot turn a short claim into evidence or replace the actual access controls for private material. Judge it by the task it helps a real consumer complete.
For Google Search specifically, the guidance is direct: llms.txt and similar special files are not used to improve search visibility. Google may crawl or index such a file as ordinary content. That is not evidence that it receives special treatment. Other agents and services must be assessed separately.
What request logs reveal
Ahrefs examined 137,210 domains using its analytics tools in May 2026. About 28% published an llms.txt file. Among domains with a valid file, 97% received no requests to it that month. Its sample is more technical and SEO-aware than the wider web.
| File request status in May 2026 | Share of domains with a valid file |
|---|---|
| No requests | 97% |
| At least one request | 3% |
Source: Ahrefs, June 2026. The base is the roughly 38,000 domains with a valid file, not all 137,210 domains or all websites.
The study also classified 96% of requests that did reach the files as bot traffic. That percentage counts requests, not files. It does not mean that 96% of files were read by AI search tools; the bot category includes other kinds of software.
These findings weaken the case for treating a file as an urgent route to AI visibility. They do not prove that no agent can use one. A month of request logs also cannot tell us how a later version of the proposal performs in a specific workflow.
About 38,000 domains with valid files in an analytics customer sample. A request is not evidence that an answer used the file. Ahrefs, June 2026.
Presence, requests and citations are different checks
SE Ranking's report found llms.txt on 10.13% of nearly 300,000 domains. Its domain-level analysis reported no useful relationship between file presence and citation frequency in the tested data. It used statistical and predictive methods, not a randomized rollout of files.
| Evidence | Result and measured object | What it supports | What remains open |
|---|---|---|---|
| Ahrefs, June 2026 | 97% of valid files had no requests in May; logs from an analytics customer sample | Check whether a proposed resource is being fetched at all | Whether a fetch changes an answer or citation |
| SE Ranking, October 2026 | 10.13% adoption; no useful citation relationship reported in its domain sample | File presence is not an established citation tactic in that analysis | A causal effect for a defined tool, site or file version |
The studies do not form a before-and-after trend. They cover different site populations and outcomes. Both vendors sell search tools, and neither can establish how every agent will behave. Read them as a reason to demand a clear use case and evidence, rather than a universal verdict on all machine-readable guides.
Where should the next piece of work go?
Use this editorial decision table to separate a useful delivery from an unsupported promise. The checks describe work you can inspect, not guaranteed ranking factors.
| Proposed task | Useful deliverable | Completion check | Outcome to measure separately |
|---|---|---|---|
| Create llms.txt for a known agent | A maintained guide to the material that agent needs | The named workflow can find and use the right material | Search citations or business results |
| Fix a public service page | A working page with readable scope and proof | The intended visitor can reach and understand it | Whether search systems index or cite it |
| Review crawler access | A record of relevant rules and actual responses | The intended crawler can fetch the intended public URL | Whether the page is selected as a source |
| Improve a weak claim | A supported answer with its conditions | A reader can trace the evidence | Whether it changes qualified demand |
Consider a fictional industrial supplier. Its audit flags a missing llms.txt file, but its main service URL returns an error and its scope document is out of date. Generating the file would leave both buyer problems in place.
The useful first job is to restore the public page and confirm the scope with the service owner. If the team also has an agent-based customer help workflow, it can test a guide for that task afterwards. The acceptance check should name the question, the correct source and the expected answer. This example shows a priority decision, not a measured client result.
llms.txt and AI Search: Common Questions
Should we delete a file that already exists?
There is no need to delete a useful, maintained resource solely because it is not a Google Search tactic. Check who uses it and what it costs to keep accurate. Google's guidance distinguishes search visibility from uses by other services.
Can llms.txt block a crawler?
Do not rely on it as an access-control mechanism. The proposal distinguishes it from robots.txt. Public crawler rules and protection of private content require their own checks.
Is a bot request a successful test?
It proves only that a request reached the recorded URL under those conditions. To test use, inspect whether the intended workflow found the right source and produced the right answer. A search citation needs a separate observation.
What evidence should an implementation proposal include?
Ask for the intended consumer, the material it needs, the maintenance owner and a concrete test. If the promised result is more AI citations, ask how that change will be distinguished from other work or normal answer variation.
Review the proposed priority
If an audit makes llms.txt urgent without explaining the buyer or agent task, bring the finding and the affected pages to SemShift. We can discuss which work removes a real barrier and which claims still need a test.