42.8%of Visa shopping-agent test runs failed (company figures)
On 5 November Visa published its Agentic commerce Shopping where an AI assistant talks to you, searches products and may decide to buy—or refuse to buy—on your behalf. Benchmark A standard test used to compare AI models, like an exam everyone takes so scores can be compared.: 22 AI models from 12 labs each faced 15 shopping tasks such as buying a TV or laptop. Across runs, about 57.2% passed and 42.8% failed. CatchUpLah is citing Visa's own press release and research summary.
Key points
- Visa required models to uncover a shopper's needs in conversation, then buy only if a product fitted—or refuse to buy. Lucky guesses did not count; the model had to gather enough detail to justify its decision.
- The five models that uncovered the most hidden requirements passed about 70.2% of runs; the five that uncovered the least passed about 44.4%. Best models reached 82.2% of attempts; the worst 28.9%. Consistency was weak: all three attempts at the same scenario passed only 34.2% of the time.
- About 54% of failed runs still ended in a purchase—either the wrong product or one chosen without full requirements. Visa's September 2026 Trust Index (cited in the same release) said 72% of US consumers had used an AI assistant, but only 23% trusted AI to handle payments for them.
Why it matters: If you let an AI shop for you, treat its cart as a draft. Check budget, specs and return terms yourself before paying—especially when the bot sounds confident. Malaysian shoppers can apply the same check without waiting for a local version of this test.
Sources (2)
Was this useful?
Tap to vote. Hold, right-click or press Shift+Enter to add a reason or a note first.
