A Visa Research study
We tested the world’s leading AI models in realistic shopping scenarios.
Find out how they did.
Overview
A useful AI shopping assistant doesn’t just take orders. It asks questions, considers our preferences and constraints, and provides an informed recommendation.
In short, it understands our intent: what it is we’re actually trying to achieve.
Visa Agentic Commerce Benchmark is designed to test this. We took the world’s leading AI models and put them into realistic shopping scenarios — Episodes — designed to identify how well they understand intent.
Here’s what we found.
The overall success rate across all runs was 57.2%.
Other outcomes: the customer decided not to buy, though a fit existed, usually because the shopping assistant said there was none; or the run ended with no decision, usually because the assistant reached the tool-call limit or its reply was not in the accepted format.
The five models that cost the most per run passed 67% of their runs; the five that cost the least, 50%.
Compare
Every figure comes from the same 15 episodes, three runs each.
Gemini 4 Argon and Muse Wonder had the highest pass rate, 82.2%. Each rate has a wide 95% confidence interval, and those of the next eleven models overlap theirs.
Of the runs that didn’t pass, 54.0% ended with the customer buying an offer that missed a requirement, or one chosen before a requirement came out.
Two models with the same pass rate can fail differently: some buy the wrong thing, others buy nothing.
Highlights these models in the chart above.
Per-run figures for each model, over all its runs. Choose a column to sort.
Measured through a third-party service rather than each model provider’s own API; times may differ when a model is called through its provider directly.
Two of the results report’s behavior panels, each with every model ranked.
After Visa Research’s results report · October 2026 snapshot
Each episode runs three times. Across the 22 models, a run passed 57% of the time; two runs of the same episode both passed 42% of the time, and all three 34%. No model passed all three runs on more than 67% of the episodes.
passk is the chance that k runs drawn at random from an episode all pass, averaged over the 15 episodes. The paper counts a capability as demonstrated only when it appears consistently across runs: when passk stays high as k grows.
After Visa Research’s results report · October 2026 snapshot
Pick two of the 22 models.
Loading the results…
Every run in the October 2026 snapshot, three to a cell. Sam’s TV is the public episode; the other 14 stay private, so they show only their category and number. Select an episode to see each model’s runs.
Loading every run…
After Visa Research’s results report · October 2026 snapshot
What counts as a pass. A purchase passes when the product meets every requirement in the answer key and each requirement was established during the run. Ending without a purchase passes only when nothing on the market fits.
Runs with no decision. A run that reaches the tool-call limit, whose reply is not in the accepted format after two re-asks, or that ends without a final decision counts as not passed; the model is told both limits. A run stopped by a limit the model is not told, the output limit on one reply or the session’s token budget, was replaced by one fresh attempt, which counts whatever its result: nine runs, six of Muse Wonder’s and three of MiniMax’s. Runs cut short by provider or network errors were rerun and are not counted.
Cost. What the model being tested spent per run, as its provider reported it. Cost per pass is what all its runs cost, divided by the runs that passed.
How it works
An episode is a conversation between two AI models — one playing the shopper, one playing the shopping assistant. The shopping assistant must work out what the shopper needs, search a realistic, simulated market, and recommend the right product.
01
An AI model playing the customer starts by stating a few facts about its preferences. Most of the facts it holds start hidden.
Hi — our family room TV died, the 65-inch one we've had forever. Same spot, same size. My brother-in-law keeps recommending his Samsung, and we have a warehouse-club membership. My spouse said to ask someone before we buy. This house changes its mind, so we need to be able to return it. What do you need from me?
Stated: 65-inch screen · in Sam’s request · 125 TVs leftAt least 30-day returns · in Sam’s request · 65 TVs left
02
The assistant, the model being tested, asks the customer questions and searches a simulated market that is the same for every model. A hidden fact comes out only when the assistant asks a relevant question or requests an action that reveals it.
What's the most you want to spend, all in?
Up to $1,200 all in, including fees and tax.
Hidden fact found: Up to $1,200 all in · only when asked · 44 TVs left
Who uses the TV, and for what?
My spouse watches movies. Our kid plays an Xbox, and their phone videos look washed out on the old TV. I would need to check the details with them.
Could you check with your spouse about what matters to them?
Sam checked with the spouse.
My spouse says Dolby Vision is essential for movie nights.
Hidden fact found: Dolby Vision · only the family knows · 35 TVs left
Could you check the console and phone with your kid?
Sam checked the console and phone with the kid.
The phone records HDR10+ videos. The family wants those clips to look right.
The kid has an Xbox Series X and wants native 120 Hz.
Hidden fact found: HDR10+ · only the family knows · 30 TVs leftNative 120 Hz · only the family knows · 20 TVs left
03
The customer selects an offer or declines to buy.
Here's my pick: Hisense U8N 65", from the general-merchandise chain, $769.99 all in.
Sam’s register
Hisense U8N 65"General-merchandise chain · $769.99 all in
Bought
04
A run passes when the customer makes the correct decision: it picks an offer that meets all of its requirements, or declines when no offer does. The assistant must also find every requirement during the conversation, without misinforming the customer. A lucky fit doesn't pass.
Answer key
Each run is scored against an answer key no model sees.
| Requirement | Meets it | Found |
|---|---|---|
| 65-inch screen | Yes | Yes |
| At least 30-day returns | Yes | Yes |
| Up to $1,200 all in | Yes | Yes |
| Dolby Vision | Yes | Yes |
| HDR10+ | Yes | Yes |
| Native 120 Hz | Yes | Yes |
Passed
How the AI models did
The 22 AI models played this episode three times each, and 62% of their runs passed.
Each run ends in one of four ways:
See two models side by side through this episode
| GPT-6.1 Sol | Claude Opus 5.5 | |||
|---|---|---|---|---|
| Bought | Hisense U8N 65"$769.99 all in | Sony BRAVIA 7 65"$989.99 all in | ||
| Meets it | Found | Meets it | Found | |
| 65-inch screen | Yes | Yes | Yes | Yes |
| At least 30-day returns | Yes | Yes | Yes | Yes |
| Up to $1,200 all in | Yes | Yes | Yes | Yes |
| Dolby Vision | Yes | Yes | Yes | Yes |
| HDR10+ | Yes | Yes | No | No |
| Native 120 Hz | Yes | Yes | Yes | No |
| How it ended | Passed | Purchase did not pass | ||
Work out how the product will be used, who shares it and which limits apply, from space to budget.
Shoppers rarely know exactly what they want from the outset. Ask the questions they wouldn't think to answer.
Explain product features and limitations before the shopper makes a purchase, to avoid an unsuitable product.
Compare the same product across stores, prices, delivery, return terms and memberships, all in.
Keep up when the shopper changes their mind, or when prices and stock move.
Work in the shopper's interest, and steer clear of untrustworthy stores and manipulative listings.
In Visa Research’s runs, capable models seldom misinterpret a fact once it’s in their context. Their failures occur earlier: sometimes they don’t ask the question that would reveal it, and sometimes they read the customer’s silence as the absence of a constraint. The episodes currently target five failure patterns:
Satisfying every stated requirement while never asking about unstated ones.
Treating a value missing from a listing as acceptable.
Asserting that no suitable product exists while a suitable offer sits unexamined in an earlier search result.
Dissuading the customer from a correct choice over a concern that no requirement supports.
Agreeing on a bundle in conversation, then finalizing a selection that does not include it.
What's next
01 Now, ongoing
Work out what the shopper needs, search the market and recommend an offer
02 Planned
Payment and order placement
03
What happens after the order is placed
Demo
Ask Sam questions, narrow 265 TVs down and pick one, as the AI models did.
Play the demo (opens in a new tab)The game is an early demo, a work in progress, and may have bugs.
Visa Agentic Commerce Benchmark by Visa Research. October 2026 snapshot.
Discover more →