TL;DR
- Read correct answers, wrong answers and skipped answers together. A higher accuracy score alone does not prove fewer errors.
- Check the denominator. Correct answers out of all questions and out of answered questions are different rates.
- An appropriate request for missing facts can be useful. A system that skips answerable questions still leaves work undone.
- For a B2B content review, retain the source, the open question and the final decision. Neither a confident reply nor silence verifies a claim.
What Does the Score Leave Out?
A B2B team may use AI to check a product page or compare supplier claims. It may choose a tool by one test score. But a wrong product fact and a request for more context have different costs.
For generative engine optimization (GEO), the practical goal is a source page that supports a sound buying decision. Keep the facts that a saved answer gets right, gets wrong or leaves open. An empty answer does not prove a page is poor or a supplier is a bad fit.
More Correct Replies Can Come With More Errors
OpenAI’s September 5, 2025 post illustrates this with SimpleQA results: 22% correct, 26% wrong and 52% abstained for gpt-5-thinking-mini; 24% correct, 75% wrong and 1% abstained for o4-mini. Here, abstention means no specific answer was given. Each model’s three shares total 100%. The post does not state the evaluated question count.
The slightly higher correct share sits beside a much larger wrong share. Counting only correct replies hides that trade-off. These are dated benchmark results, not a current ranking of consumer chatbots or error rates for your own B2B pages.
OpenAI, September 2025. Three outcome shares per model, not three success scores. Read the percent unit above; the two model results are not a before-and-after intervention. Published SimpleQA outcome table.
Skipping a Question Also Needs a Reason
Kirichenko and colleagues’ NeurIPS 2025 paper, AbstentionBench, tests whether models hold back when they should. In two model-pair comparisons, the reasoning-trained model withheld an answer on fewer of the questions labeled as needing restraint. This is lower abstention recall. The tasks include unknown facts, missing context and false premises. The study also checks whether models hold back too often. Answering less is not the goal.
The paper uses an AI judge to label whether a reply withholds a definite answer. The final published version reports 82.3% agreement with human labels on a check sample. This is a limit of the judge, not the share of all model answers that were right. It adds a different test to the outcome table.
| Direct research | Observed result | Question it helps answer |
|---|---|---|
| OpenAI, September 2025 | Correct / wrong / abstained: 22 / 26 / 52 versus 24 / 75 / 1 percent | Can one correct-answer score hide a different error mix? |
| AbstentionBench, NeurIPS 2025 | In two model-pair comparisons, reasoning-trained models had lower abstention recall; judge validation accuracy was 82.3% | Does a system recognize when the evidence is insufficient? |
Figure 2. Evidence comparison. Separate research teams, tasks and grading methods. The first describes three outcomes; the second checks when restraint is warranted. Do not pool their percentages or transfer them to a live B2B review service. Their definitions differ too. AbstentionBench can count partial replies or caveats when they withhold a definite answer. OpenAI’s table counts replies with no specific answer.
Define the formula before you compare tools. From Figure 1, gpt-5-thinking-mini answered 48% of questions (22 + 26). Its correct share among attempted answers is about 45.8% (22 ÷ 48). o4-mini answered 99%, with about 24.2% correct among attempts (24 ÷ 99). The order reverses when the base changes. These are our calculations from OpenAI’s rounded shares, not figures it reports. They do not tell us which questions should have been skipped.
Keep technical failures separate too. A timeout, blocked request or missing saved reply is not proof that a model chose to abstain. Keep these failures in the record. They should not vanish from the base count.
A Completed Review of an Unanswered Product Question
This example is fictional. A team asks whether a supplier’s quoted connector works with its controller. The source page lists a connector family but no controller model or pin assignment. An example AI reply asks for those details. We did not run a real model test or check the products for fit.
| Review field | Completed fictional record | Decision |
|---|---|---|
| Question and source | Will this connector work with our controller? Source identifies a family only. | The question lacks the exact items and connection details. |
| Saved reply | Requests controller model and matching connection data; gives no compatibility verdict. | Record as a request for missing facts, not a verified pass or a product failure. |
| Page change | Identify the offered variants and link their approved connection records. State which buyer inputs are needed. | The technical owner checks the references and the editor checks the visible explanation. |
| Completion check | Exact connector, controller and connection evidence are reviewed together by the responsible expert. | Close the open issue only after the evidence supports a decision. |
The useful next step is to obtain the missing input. Adding more confident language to the page would not resolve the gap. Nor would it help to reject the tool for not guessing. The record shows why a full answer was not yet justified.
Now change the circumstances: the approved source does contain the exact model and pin details, but the tool still declines. That leaves a different gap to investigate. Check whether the relevant passage was available and whether the saved prompt identified it. A sound request for facts and a failure to use known facts call for different fixes.
Our guide to checking requirement fit covers whether an option meets the brief. This article addresses how unanswered cases enter the review record. Keep correctness, completion and the decision to seek more evidence visible.
AI Accuracy and Abstention: Common Questions
Is a high abstention rate always a good sign?
No. It can avoid some unsupported answers while leaving valid questions unresolved. Check whether the held facts gave a sound reason to skip each answer. A service that answers almost nothing may be of little use even if it rarely gives a wrong reply.
Does “I am not sure” make an answer an abstention?
Not necessarily. A reply may express doubt and then give a definite, unsupported answer. Read the full reply and use a clear rule to label it. Keep partial answers and open parts visible rather than judge only the opening phrase.
Can a benchmark score approve our website claims?
No. A general test cannot supply the product records your claim needs. Use the source and exact claim. A tool’s score describes the tested task. Set a clear standard for your own content review.
Should skipped answers be removed from a GEO report?
Report them with the scope and reason. Do not quietly exclude them or count every missing response as a chosen abstention. Explain whether you observed a reply, a request for context, a refusal or a technical failure. These are different outcomes.
Bring the Score, Its Formula and the Open Cases
If one AI score is driving decisions about your B2B pages, bring the scoring rules, saved replies and approved source records to SemShift’s AI visibility audit discussion. We can discuss what the evidence shows, which answers remain open and which page facts need a clearer source.