TL;DR
- Treat a single answer as an observation. A change in list position or wording can occur without a change to your website.
- Compare the same buyer needs, platforms and counting rules across repeat samples. Keep answer accuracy separate from presence.
- Ask for the saved answers and the denominator behind a visibility percentage. Failed checks and missing answers need their own record.
- Repeatable presence is useful evidence, but it does not by itself prove that an edit caused growth or that the audience represents your buyers.
The wording can change while the answer stays similar.
Ahrefs’ November 2025 study tracked over 43,000 keywords for a month. Each had at least 16 recorded AI Overviews. The text changed in 70% of consecutive comparisons. On average, 54.5% of cited URLs overlapped. Yet meaning stayed close: the semantic similarity score was 0.95. That score does not mean the words or sources were the same. Checks were not daily.
Those are separate questions for a marketing team. Did the system change its words? Did it choose a different source? Did it change what it said about a supplier? Your response should depend on which event occurred.
A buyer may still receive sound advice when a source rotates. Yet a stable answer can also repeat an outdated service claim. Neither the presence of change nor the absence of change tells you whether the business is represented well.
Ahrefs, November 2025. The second value is 100 minus the reported change rate. This is text change, not brand loss or a daily probability. Study and method.
Changing lists do not make every measurement useless.
SparkToro and Gumshoe’s January 2026 research collected 2,961 responses from volunteer runs across 12 prompts and three AI tools. Lists and order varied, but some brands appeared often: one agency appeared in 85 of 95 Google AI responses for a specific recommendation task. That is frequency within one test, not market share.
This matters when someone sells you an “AI rank.” A firm listed first once may be absent from most other answers. Another may appear often but in a different place. Change the buyer’s use case, location or budget, and the list may change too.
Ask what the report is meant to show. Being present often for a set of buyer needs is a more useful goal than holding one place in a list. The questions still need to match the market you serve.
Two studies, two kinds of instability.
Both studies give reasons to repeat a check. Neither sets a sample size that works for every task. Nor do they show that a tool can capture what every real buyer sees.
| Study | Reported result | What was examined | What the result cannot establish |
|---|---|---|---|
| Ahrefs, 2025 | 54.5% citation overlap; semantic similarity 0.95 | Repeated Google AI Overviews | How any particular B2B brand performs today |
| SparkToro / Gumshoe, 2026 | 85 appearances in 95 responses in one example | Repeated recommendation prompts; volunteer collection | The distribution of real buyer prompts or a universal stable ranking |
The studies examine different tasks and products. They are not two estimates of the same error rate. SparkToro also worked with an AI tracking vendor; its article explains that relationship. Keep the method visible when using the findings to choose reporting requirements.
Decide whether the change calls for an edit.
Suppose a report shows your company dropping out of one answer after a capabilities-page update. Rolling back the page immediately would assume that the edit caused the loss. First ask whether the page is still accessible and whether its approved facts remain correct.
Separate a site fault from a change in the AI answer. Use the table to choose what to check. None of these steps promises to bring a source link back.
| Observation | Next check | Reasonable action |
|---|---|---|
| Brand missing once; page and facts unchanged | Repeat the recorded question and inspect the cited sources | Record the event; do not rewrite the page on this basis alone |
| Page returns an error after a release | Check the published URL and relevant access logs | Repair the confirmed access fault, then retest it |
| The answer repeats the wrong service region | Compare the cited page and the approved region list | Correct facts you control; request corrections from outside publishers |
| Presence falls across several comparable samples | Check changes in prompts, product mode, competitors and source selection | Investigate a defined topic before expanding the scope |
| Presence rises but inquiries do not | Review landing-page fit and the inquiry records | Keep visibility and lead quality as separate results |
A better way to describe a small improvement.
Illustrative example — fictional data. A machining supplier is reviewed against a fixed set of application questions. Its first sample has 40 usable answers, with the brand present in 12. A later comparable sample has 40 usable answers, with the brand present in 16. The observed rates are 30% and 40%.
That is a rise of 10 percentage points, or about 33% relative to the first rate. It does not prove that the website work caused the change. The sample is small. The sources and the AI product may have changed too.
Now suppose the four added mentions describe the firm as a reseller when it manufactures parts. The higher presence rate has exposed a fact problem. It is a reason to review the source trail, not a result to celebrate without qualification.
Keep both samples and flag the wrong descriptions. Log the site edits made between the tests. If you add questions for a new market, report that set on its own. Do not splice its rate into the old trend.
A report note that keeps the evidence intact
Question set: machining applications, same market and language
Usable answers: 40 before / 40 after
Brand present: 12 before / 16 after
Observed rate: 30% → 40% (+10 percentage points)
New concern: 4 later answers misdescribe the business
Next action: inspect linked sources and correct approved business facts
Conclusion: observed change; cause and commercial impact not establishedAsk for a record you can inspect.
Ask to see the answers behind the dashboard. A useful record includes the question set, platform, mode, dates and counting rule. Keep failed checks and missing answers in the record too. Decide how to treat them before you compare periods.
Also ask how the questions were chosen. A neat set of prompts written by a marketer can be useful for repeat checks, but it is not a record of actual demand. Sales questions, search queries and customer interviews can improve that set without making it a census.
Use the AI visibility measurement guide for terms and formulas. Here, the task is to judge whether a change is worth acting on. A new result does not always call for a new tool.
Questions before you trust an AI visibility trend.
How many times should we repeat a prompt?
There is no universal count established by these studies. Set the number before testing, based on the importance of the decision, available capacity and observed variation. A larger sample cannot fix an irrelevant question set. Be cautious about narrow changes supported by few answers.
Can an agency use screenshots as evidence?
Yes, as records of specific answers with their context and date. Screenshots become misleading when presented as a stable rank or selected from many unreported attempts. Ask for the full run record behind a headline result.
Should we change content after every lost citation?
Repair a confirmed error promptly. For a lost citation without a known page or fact problem, inspect repeated observations and competing sources first. Frequent blind edits make it harder to learn which change helped.
Does a stable answer prove that its claims are accurate?
No. Consistency and accuracy are separate checks. Compare important company claims against approved business records, then investigate the sources that appear to support the answer.
Bring the change you need to explain.
If your team has conflicting screenshots or a sudden dashboard shift, bring the saved answers, target questions and recent website changes. SemShift’s audit can help separate an access or fact problem from a measurement question and identify what to investigate next.