TL;DR
- Read the task, tool, people and time period behind an AI speed claim before applying it to your offer.
- A timed experiment and a user’s sense of time saved are different forms of evidence. Keep them separate.
- Historical studies found gains in one setting and slower work in another. Later updates matter; neither result forecasts today’s whole market.
- State what the evidence measured. Use a local evaluation to check the buyer’s workflow rather than promise a team-wide gain from a borrowed headline.
A product claim needs a defined task
For generative engine optimization (GEO), the source page should give a buyer facts they can check. “Faster coding” leaves open what was timed and what passed review. A buyer needs that scope before using an AI summary to build a business case. This article concerns the evidence behind the claim, rather than which coding tool to buy.
One Coding Task Has a Narrow Finish Line
GitHub’s study report, first published in 2022, describes 95 developers assigned to two groups to build an HTTP server in JavaScript. Reported mean completion times were 71 minutes with Copilot and 161 minutes without it. The report calls this about 55% faster. The paper’s Results section makes these time means conditional on completing the task. They are not means across all 95 recruits. This is a task result, not a measured gain in a full delivery cycle.
The task had a defined finish line. Passing it is useful evidence about that task. It does not establish how much work a team can ship after setup, review and release. Read the source’s completion rule before turning a time figure into a capacity claim.
GitHub’s 2022 report, rounded means among task completers; 95 people were recruited. These are different groups, not a before-and-after change in the same developers. The paper reports 160.89 and 71.17 minutes. Study paper, Results and Figure 6.
Belief, Task Time and Team Output Are Different Measures
METR’s July 2025 trial involved 16 experienced open-source developers and 246 real issues in repositories they knew. Issues were assigned to allow or disallow AI. With early-2025 tools, AI-allowed work took an estimated 19% longer. Developers had expected a 24% time reduction; after the work, they still estimated a 20% reduction.
The three numbers answer different questions. Two describe beliefs about time saved. One estimates the effect of allowing AI in that study. The gap is a reason to check a claim’s measure, rather than a reason to dismiss every user’s experience.
These findings need not cancel each other out. Building the same new server and making changes in a familiar, complex repository place different demands on a developer. The tools and dates differ too. Neither comparison isolates which of those differences explains the gap.
Our reading is that a useful claim has to keep its setting attached. Averaging the headlines would not produce a sound forecast for a third team. Nor can a reduction in one task’s time be relabeled as an equal rise in features shipped, revenue or team capacity.
| Direct research | Actual result and scope | Limit on a public claim |
|---|---|---|
| GitHub report, 2022 | Mean times among completers: 71 vs 161 minutes; 95 recruits; one JavaScript server task | Does not measure an entire team’s delivery output |
| METR trial, 2025 | Estimated task time +19%; expected −24%, perceived −20%; 16 developers, 246 issues | Belief is not measured gain; result is specific to early-2025 tools and tasks |
| METR update, 2026 | Later estimates affected by who joined, which tasks they submitted and how time was recorded | Does not supply a reliable current gain to substitute for either older headline |
Figure 2. Evidence comparison. The table compares actual results and methods; the studies are not competing estimates of one shared population. GitHub sells the product examined. METR’s later update comes from the same research group as its earlier trial, so it is follow-up evidence, not a third independent replication.
The Later Update Changes How the Old Result Should Be Used
METR’s February 2026 update says its later experiment gave an unreliable signal of the current effect. Some developers did not want to work without AI. Some held back tasks they expected AI to help with. Parallel agent use also made time reports harder to interpret. METR believed gains had likely improved, but called its evidence for the size of that change very weak.
That update blocks a simple “AI still slows developers” claim based on the older result. It also does not justify replacing the old number with a firm new promise. A dated result can remain useful evidence about a particular test while ceasing to be a good description of current use.
This is why a source note should include the study date as well as the date the page was checked. Updating a page’s timestamp cannot update the tools or people in the experiment. When a supplier cites research, ask whether later work from that source changes the proposed wording.
Turn the Research Into a Claim a Buyer Can Inspect
The following is an editorial review aid for a fictional developer-tool vendor. Its team has a public study but has not measured its own product’s effect on customer delivery. No figures below are client results.
| Review field | Completed fictional record | Publishing decision |
|---|---|---|
| Proposed headline | “Ship 55% more every sprint with our assistant.” | Hold: neither sprint output nor this product was tested |
| Evidence available | GitHub’s server-task study; 71 vs 161 minutes among completers | Keep the original tool, task and study attribution |
| Evidence missing | No test of this assistant, customer workflow or whole delivery cycle | Do not infer a customer gain |
| Revised claim | “Our evaluation lets your team test the assistant on an approved coding task. Compare time and accepted work with your existing approach.” | Use only if the actual evaluation offer supports this scope |
| Owner and completion check | Product owner checks the offered access; editor checks every numeric claim against its source | Pass when the page and offer agree and no borrowed effect becomes a product promise |
The change is useful even without a new percentage. It gives the buyer a testable next step and removes an unsupported output promise. Keep the study nearby as context if it helps explain why the team should measure the task. Do not style it as a badge proving this product’s performance.
For a local test, define what counts as accepted work and what time is included before it starts. Keep setup, review and rework visible. Record failures and abandoned tasks as well. This is our suggested check, not a validated universal trial design. Small local results should retain their own limits.
The distinction also helps an AI answer quote the page accurately. A sentence naming the tested task and scope is more useful to a buyer than a bare “up to” number. It does not guarantee a citation. Our demo and trial guide covers how to describe the access a buyer actually gets.
AI Coding Claims: Common Questions
Can we still cite an older study?
Yes, when it is relevant and clearly dated. Name the tested task and tool. Check later updates and explain any limit that changes the claim. Do not present the study as a current test of your product.
Does “up to” fix a claim that exceeds the research?
It does not establish missing evidence. A maximum still needs a defined measure and basis. If the source timed one task, adding “up to” cannot turn it into proof of whole-team output.
Should we measure only time?
Time may answer a narrow question. A buyer can also need to know whether the work meets an agreed standard and how much review it takes. Choose measures that match the claim and retain results that reveal a trade-off.
Does a clearer evidence note prove better AI visibility?
No. It makes the claim easier for a reader to inspect. Citation frequency and business outcomes require separate measurement. This article reports neither a visibility lift nor a customer productivity gain.
Review the Claim Before It Becomes a Forecast
If your product pages turn research headlines into promises about customer output, bring the pages, cited studies and approved evaluation scope to SemShift’s GEO service discussion. We can discuss which claims need narrower wording, stronger proof or a clearer path to a buyer’s own test.