TL;DR
- Save the order of vendors with each AI comparison. The list is part of the test input.
- Models and tasks show different position preferences. There is no slot that always wins.
- An answer can switch between two close options. A switch alone does not prove that the new choice is worse.
- Try the same facts in more than one order, keep repeat runs and check the reasons against the brief. Do not select just a favorable screenshot.
Why the Candidate List Belongs in the Record
UC Irvine’s August 2026 research news post describes models changing choices when the same options are reordered. It raises a useful question for a B2B review. Did the facts change, or did only their order change?
For a generative engine optimization (GEO) report, save the full prompt when a team supplies vendors and asks AI to choose. The result concerns those supplied options. It does not show how often the service would find your brand on its own.
First Position Was Not Favored by Every Model
A University of Granada preprint, October 2026, tested product choices with 2,000 generated queries and 20 random list orders per query. In its five-option test at temperature 0, GPT-OSS 20B chose the first slot 25.38% of the time and the fifth 17.28%. LLaMA 4 Scout showed the reverse: 17.11% first and 20.99% fifth.
For GPT-OSS 120B, the shares were 23.18% first and 17.93% fifth. Each share averages over queries and random list orders, using valid responses. It is not a named vendor’s chance of winning. With five slots, the no-position-preference reference is 20%.
Peláez-González et al., October 2026 v1, Table 5; p(i) × 100. Two of five positions shown. Fixed product-catalog prompt, 2,000 queries × 20 orders; parsing errors and refusals excluded. The 20% reference is not a brand success target. Study, method and Table 5.
An Order Effect Is Not Always a Worse Choice
Yin, Vardi and Choudhary’s August 2026 PNAS Nexus study compared résumés and paint colors. In three-option tests at temperature 0, GPT-4o-mini chose the middle option for Plain colors 100% of the time. Claude 3 Haiku chose the middle for Harsh colors 83% of the time. These are different quality tiers, not a model ranking.
The paper separates fragile preferences from robust choices. For a buyer, that raises a further question. Did the answer switch between close options, or did the new choice fail the brief? A switch and a worse choice need separate proof.
| Direct research | Reported result | Boundary for a B2B review |
|---|---|---|
| Granada, October 2026 | GPT-OSS 20B: first 25.38%, fifth 17.28%; five-candidate test | Marginal position shares; supplied retail candidates, not open-web brand visibility |
| PNAS Nexus, August 2026 | Middle choice: 100% GPT-4o-mini/Plain; 83% Claude 3 Haiku/Harsh | Different color tiers; not a supplier fit or purchasing study |
Figure 2. Evidence comparison. These are independent teams and different experiments. One uses catalog recommendations; the other studies preference structure. Neither establishes a B2B conversion effect or a dependable website ordering tactic.
The Granada study also notes that order can break ties among close options. Its measures of worse choices have further limits. Do not treat every changed answer as proof that an AI chose an inferior supplier.
A Completed Review of an Order-Sensitive Comparison
This example is fictional. The outputs below show how a review record can work; we did not run this AI test. A team compares calibration services for one gauge type in one region. The same approved facts are present for suppliers A, B and C. Current quotes and availability are not yet checked.
| Review field | Completed fictional record | Decision supported |
|---|---|---|
| Fixed task and evidence | Same gauge type, service region and supplier fact cards in both prompts | Keep the comparison scope unchanged |
| Input change | Prompt 1 orders A–B–C; prompt 2 orders C–B–A; whole cards move intact | Order changed; descriptions and instructions did not |
| Illustrative outputs | First answer chooses A; second chooses C; both cite regional coverage | A discrepancy to investigate, not proof of a better supplier |
| Next check and report | Repeat the original order as well as reordered lists; retain all usable answers and failures | Do not claim a stable preference from one pair; verify fit and quotes separately |
Move each whole fact card. If you shorten one card or add a stronger claim, you change the facts as well as the order. Keep each supplier’s ID fixed. Avoid adding labels such as “best” to one version. Those edits would change the test.
Repeat runs with a fixed order help show normal answer variation. Runs with a changed order add a different check. A small test can reveal a gap worth a closer look. It cannot measure a general position effect. Report the saved cases and their limits.
A supplier can make its facts clear and easy to check on comparison pages. These studies do not justify hidden instructions or a promise to control source order. Our comparison-page guide covers evidence for buyers.
Keep this check separate from requirement fit. A stable answer can still miss a must-have. A changing answer can still name two options that both fit. Those are different findings.
AI Candidate Order: Common Questions
Does choosing the first option prove position bias?
No. The first option may be the best fit. Keep the facts and task fixed while changing the order, and account for repeat-run variation. One choice cannot separate these causes.
Should every vendor be recommended equally often?
Not by name. A better fit may deserve more recommendations. The 20% reference concerns slots after five options are put in random order. It is not a rule that all five vendors deserve equal preference.
Can reversing a list remove the problem?
A reversal can reveal a gap, but it is not a cure for every case. Other slots and normal answer variation can still matter. Match the check to the decision. Two outputs are not a statistical audit.
Does this mean a website should put its brand first?
No. These tests put the options inside prompts. A live service may find and arrange its own sources. The studies do not show that moving a website section changes how often a brand is recommended.
Bring the Prompt, Candidate Cards and Saved Answers
If a supplied-vendor comparison is used as proof of AI visibility, bring the full prompt, fact cards, list order and saved answers to SemShift’s AI visibility audit discussion. We can discuss what the test shows and what evidence would support a narrower claim that can be checked again.