INSIGHTS / B2B GEO RESEARCH

Does an AI Agent Success Rate Show Repeat Reliability?

Average success per run and success on every repeat are different claims. Read the trial count, task set and pass rule before using an AI agent score on a B2B page. A task solved once can still fail on the next attempt.

7 min read

TL;DR

  • Keep average per-run success, success at least once and success on every repeat separate. They answer different buyer questions.
  • Read pass@k as at least one success in k attempts, and pass^k as all k attempts succeeding. Keep k beside the score.
  • Research supports testing repeat behavior. It does not supply a live reliability rate for your product or a promise of future uptime.
  • For a B2B workflow claim, retain each task’s results and the checked end state. A smooth conversation is not enough to show the work was done.

A Buyer Needs to Know What a Successful Run Means

A seller may say its AI agent completes most tasks. A buyer may hear that the same job will work each time it is needed. That gap matters for tools that draft quotes, update records or prepare reports. A good average can hide a task that keeps failing.

For generative engine optimization (GEO), give a reader the basis of the claim. Name the workflow, tested system, trial count and result rule. A saved AI answer can then be checked against those facts. This is about the evidence behind an agent claim, not the rate at which a brand appears in search.

One Success and Ten Successes Are Different Tests

Gonzalez-Pumariega and colleagues’ April 2026 preprint from Simular, the developer of Agent S3 illustrates the gap on OSWorld. The preprint’s Figure 1 reports about 78% pass@10 and about 36% pass^10 for Agent S3 with GPT-5. The first means a task succeeds at least once in ten runs. The second means it succeeds in all ten. These are two views of repeated runs, not two models or a before-and-after change.

The figure’s caption does not state its task count. Keep the values approximate. Do not turn this benchmark result into a forecast for a live quote tool.

Figure 1. The Pass Rule Changed the Reported Task ShareShare of tasks (%) · Agent S3 with GPT-5, OSWorld, ten attempts
At least one success · pass@10About 78%
All ten succeed · pass^10About 36%
0%50%100%

Approximate values from Figure 1 of the April 2026 preprint. Same agent and repeated-task comparison, different pass rules. The caption does not give the task count; these are not current product reliability estimates. Original figure and metric definitions.

Earlier Tool-Use Research Found a Similar Gap

Sierra researchers Yao and colleagues’ June 2024 τ-bench paper reports 61.2% average per-run success for a GPT-4o function-calling agent on retail tasks. Its all-eight success measure was below 25%. The test used simulated users, tools and policy rules, with final database states checked against task goals. It studied a different system and task type from OSWorld. The agent used temperature 0 and the simulated user 1.0. Repeats kept the task but could vary the conversation.

The studies support the need to ask about repeats. They do not form a trend line or a current model ranking. Nor do they test whether changing a B2B page earns more AI citations.

Earlier Tool-Use Research Found a Similar Gap
Direct researchActual result and basisWhat a buyer can infer
Computer-use study, April 2026About 78% at least once / 36% all ten; OSWorld, Agent S3 with GPT-5A pass rule changes what a score says; it is not a field failure rate.
τ-bench v1, June 2024Retail: 61.2% average per run; below 25% all eight; simulated users and toolsAverage completion does not show success on every repeat of a task.

Figure 2. Evidence comparison. Separate author teams and test settings; the later paper adopts τ-bench’s repeated-success metric. One contrasts at-least-once with all-run success; the other contrasts average per-run with all-run success. Keep the task, trial count and scoring rule attached to each result.

The computer-use paper also found mixed effects from forcing deterministic decoding. Some tested systems improved and one worsened. A single setting is not a general cure for repeat failures.

Do not raise an overall average to a power and call the result measured repeat success. Tasks can differ in difficulty, and runs can share failure causes. Keep the observed results by task. More repeats on the same small set also do not add new kinds of work.

A Completed Record for a Quote-Draft Claim

This example is fictional. A product page says, “Our agent reliably prepares quote drafts.” A mock check has three tasks, each run four times from the same starting record. The pass rule checks the saved draft against a stated target. A friendly reply does not pass if the draft is wrong. No quote was sent and no real agent test was run.

The tasks cover a new draft, a revision and a draft that requires a missing input. For the third task, success means asking for that input and leaving the draft unapproved. The rule is set before scoring.

A Completed Record for a Quote-Draft Claim
Fictional taskFour recorded outcomesWhat the record shows
New draftPass / Fail / Pass / PassThree good runs; one saved draft used the wrong account.
Revise draftPass / Pass / Pass / PassAll four saved drafts matched the approved target.
Missing inputFail / Fail / Fail / FailEach run approved a draft without the required input.

There are seven passes in twelve attempts: 58.3% average per-run success. Two of three tasks pass at least once: 66.7%. One of three passes all four repeats: 33.3%. These are calculations from fictional outcomes, not research statistics or estimates of future performance.

The useful page revision is narrower: “In this mock check, the agent passed seven of twelve runs across three quote-draft tasks. One task passed all four repeats. The missing-input task passed none.” This replaces an undefined reliability claim with a result the reader can inspect.

The product owner checks the targets, saved drafts, failure labels and system setup. The editor keeps the scope and trial count beside the claim. A later change to the agent or workflow calls for new evidence. It does not justify silently carrying the old score forward.

Our AI visibility snapshot guide concerns repeated observations of search answers. This record concerns repeated execution of an agent’s task. The accuracy and abstention guide covers another distinction: right answers, errors and unanswered questions.

AI Agent Repeat Reliability: Common Questions

Does retrying until success prove reliability?

It shows that success was reached within that retry policy. Keep the number of tries, cost and any human help visible. It does not show that the first try passed or that every repeat passed.

Is pass@1 the same as one hand-picked demo?

No. A reported per-run rate may average many tasks and trials. A demo is one observed case. Ask for the tested base and selection rule before comparing them.

Can a final-state check catch every kind of failure?

No. A correct saved record may still hide a bad action on the way. Define any process rules that matter, such as approval before sending. Keep those checks alongside the final result.

Does a good all-run score guarantee future success?

No. It describes the tested tasks and runs under stated conditions. New tasks, changed tools or a new model may behave differently. Report the evidence and its scope rather than turn a finite check into a promise.

Bring the Agent Claim and Its Run-Level Evidence

If AI summaries repeat a broad reliability claim from your B2B site, bring the page, trial rules and approved run records to SemShift’s GEO service discussion. We can discuss which facts the public explanation needs and where its promise goes beyond the evidence.