On November 5, 2026, Visa published the first results of its Agentic Commerce Benchmark. 22 AI models from 12 labs ran 15 shopping tasks, 3 attempts each. They failed 42.8% of runs, and about 54% of the failed runs still ended in a purchase.
We read the release next to 2 other documents from the same 3 weeks: Walmart's October 21 rules for outside shopping agents, and Shopify's November 3 earnings call. Together they describe one chain. Agent orders are arriving. A measurable share of agent decisions miss what the customer wanted. The retailer takes the product back.
What Visa measured
The Visa release sets a strict bar. The customer does not state every need upfront. A run passes only if the model discovers every requirement defined for the task and then makes the correct purchase or no-purchase decision.
The overall pass rate is 57.2%. The best model passed 82.2% of attempts, the weakest 28.9%. Across all models and scenarios, all 3 attempts passed 34.2% of the time. The release does not name the models.
Our arithmetic turns the percentages into counts. 22 models, 15 tasks and 3 attempts make 990 runs, 45 per model. The best model failed 8 runs of 45. About 424 runs failed, and about 229 of those ended in a purchase. That is about 23 runs in 100 where the model bought after missing a requirement.
A second reading sits in the 34.2%. If every attempt were an independent draw at 57.2%, 3 passes in a row would occur 18.7% of the time. Visa observed almost twice that. Results depend on which model meets which task. A dependence of that kind can be found, by agent and by product category.
Visa adds a cost line. The 5 models with the highest cost per run passed 67.1% of runs, the 5 least expensive 50.2%. The gap is 16.9 points.
What Walmart and Shopify disclosed
Walmart's October 21 post introduces Agent Profiles, the Walmart Agent Gateway and terms for AI agents. "Customers must approve agent purchases by default." A customer can instead set a spending limit, "for example, $50, $100 or $200."
One sentence in that post allocates the cost of an error. Walmart writes that its "returns process applies," and that this holds "including if an agent doesn't act as intended."
On Shopify's third-quarter call, President Harley Finkelstein said AI traffic is "up 2.6x year-over-year" and orders are growing "at 3.3x." He called the quarter "the first time we saw shoppers experimenting with delegated purchasing through personal agents," and added that volume "is, of course, small." Shopify gave no share of its $119 billion of quarterly GMV.
CFO Jeff Hoffmeister said of agent-driven sales that "the economics are exactly the same." For a platform paid on each transaction, that holds. For the merchant, it holds only if the return rate is the same. No party has published that number.
An expert objects, and correctly. The benchmark grades a product decision in a test, not a placed order. Visa says it "will extend testing to checkout, payment, and order placement." A shopper may keep a product that misses one preference. Visa also sells agent payment products, which it describes as separate from the benchmark. We do not convert 23 in 100 into a forecast return rate.
The agent's developer picks the model and pays for the tokens. The retailer takes the product back. Hikari Blue · operator note
Where the value sits
The channel converts. Finkelstein said shoppers arriving from AI channels "are converting 64% more often." The value of that conversion is net of what comes back. The National Retail Federation estimated that 19.3% of online sales would be returned in 2025.
Our reading, not a statement by Visa or Walmart: the party that chooses the model is not the party that processes the return. A developer that saves on inference moves toward the 50.2% group. The 16.9 points it gives up appear, if anywhere, in a retailer's returns line.
The lever a merchant controls is product data. In Visa's test, the 5 models that uncovered the most hidden buying factors passed 70.2% of runs, the 5 that uncovered the least 44.4%. Finkelstein said "Catalog powered searches are converting at 2x the rate of general AI search." An agent that reads compatibility, format and dimensions as structured fields has fewer requirements to guess.
What management decides now
First, make agent orders a reported channel. Tag each order with the agent that placed it. Walmart announced the customer-side version on October 21, due within a month: each agent gets a dedicated profile, and its purchases appear separately.
Second, measure return rate, contact rate and dispute rate per agent, against the site baseline, by category.
Third, set the gate by category. Keep approval on each purchase where a missed requirement is costly to reverse, such as electronics and appliances. Allow a spending limit on repeat baskets. Walmart keeps a reserve: it "may require additional approvals even for purchases within an agent's spending limits."
Fourth, read the agent terms as a cost allocation. Ask counsel who pays when an agent buys the wrong product, and whether the cost is recoverable from the agent's developer. The NIST AI RMF, GOVERN 6.1, asks for policies that address AI risks associated with third-party entities.
What could prevent value capture
Volume comes first. Shopify calls delegated purchasing small. A return rate per agent stays noisy for several quarters. The tag still has to exist before the volume does.
Identification is partial. An agent that shops through the customer's browser session looks like the customer. Only a declared channel, a gateway or a profile, yields an order that can be attributed.
Gates cost conversion. Each approval step gives back part of the gain that made the channel attractive.
What is the return rate of orders placed by AI agents on our store, by agent, and who in the company owns that number?
Track the share of orders that carry an agent identity, and their return rate against the site baseline, by category. Outside the company, watch Visa's next round, with checkout included, and the first disclosure of agent order volume by Shopify or Walmart.
Thesis. A benchmark grades the model, but the retailer absorbs the purchase, so the control belongs on the order: an identity per agent, an approval gate by category, and a return rate by agent. Govern the action, not the model
- Visa Inc. (November 5, 2026). Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail. Primary source for: 22 models from 12 labs, 15 shopping challenges, 3 attempts per scenario; overall pass rate of 57.2% and failure rate of 42.8%; best model 82.2%, weakest 28.9%; all 3 attempts passed 34.2% of the time; about 54% of failed runs ended in a purchase; 67.1% for the 5 highest-cost models and 50.2% for the 5 least expensive; 70.2% and 44.4% by discovery of hidden buying factors; pass criteria; planned extension to checkout, payment and order placement; Visa Intelligent Commerce described as separate. usa.visa.com, press release
- Walmart Inc. (October 21, 2026). Walmart Makes It Possible for Agents to Shop on Behalf of Customers, While Keeping Customers in Control. Primary source for: Agent Profiles, Walmart Agent Gateway (beta for companies and developers), terms for AI agents; approval of agent purchases by default; spending limit examples of $50, $100 or $200; additional approvals within spending limits; returns process applies if an agent does not act as intended. The Walmart Agentic Access Terms that the post refers to sit behind a verification page we could not read; we cite the post only. corporate.walmart.com
- Shopify Inc. (November 3, 2026). Third quarter 2026 earnings conference call, transcript published by The Motley Fool. Source for the statements of Harley Finkelstein, President (AI traffic up 2.6x, orders at 3.3x, converting 64% more often, Catalog powered searches at 2x, first delegated purchasing through personal agents, small volume) and of Jeff Hoffmeister, CFO ("the economics are exactly the same"); GMV of $119 billion. fool.com, call transcript
- National Retail Federation and Happy Returns (October 15, 2025). 2025 Retail Returns Landscape. Source for: an estimated 19.3% of online sales returned in 2025; total retail returns projected at $849.9 billion in 2025. Estimates and projections, from a survey. nrf.com
- National Institute of Standards and Technology (January 2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100‑1, GOVERN 6.1. nvlpubs.nist.gov, NIST AI 100‑1
- Our arithmetic, on Visa's reported inputs. Runs: 22 × 15 × 3 = 990, 45 per model. Best model: 37 of 45 = 82.2%, 8 failures. Weakest: 13 of 45 = 28.9%. Failed runs: 42.8% × 990, about 424. Purchases after a failed run: about 54% × 424, about 229, or 23.1% of 990. Independent draws: 0.572 × 0.572 × 0.572 = 18.7%, against 34.2% observed. Cost gap: 67.1 minus 50.2 = 16.9 points. Visa gives 54% as an approximation, so 229 and 23.1% are approximations.
The Hikari Blue team · Austin, November 2026
