The Real Test for Ecommerce AI Agents

How it works

One rubric, applied the same way to every vendor.

We measure two jobs separately: a Shopping Assistant that helps a shopper find and buy, and a Support Agent that resolves shipping, returns, and policy questions without a human. Every agent gets the same questions, under the same conditions, scored the same way.

Same test for all

One standard set of questions, adapted only to each store's catalog.

Agent-initiated handoffs

The auditor never asks to speak to a human. Every handoff is initiated by the agent itself because it didn't have the information needed to answer.

Cold sessions

A fresh browser with no memory on every run, so nothing is warmed up.

Blind scoring

Names hidden, and every claim quoted from the transcript before it counts.

The numbers

How every vendor performs, on both jobs.

Each vendor gets two scores, one for each job. Automation measures how many conversations are resolved without a human. Quality is a blind 0–100 score of how well the agent sells or resolves. Speed measures how long it takes to reach a complete answer.

Overall position, last 120 days.

Shopping and Support are each a 0 to 100 composite: automation, quality and speed, weighted per job (shopping 40/35/25, support 50/40/10). Overall is the average of the two. Click a column to re-sort.

Ranking requires a minimum sample. A vendor must record at least 15 judged conversations in a given job (roughly three storefronts) to qualify for a head-to-head rank, so that no single storefront can distort the result. Vendors below this threshold remain visible in the profiles but are not ranked. Currently unranked on sample size: none. One misconfigured Shopping deployment is excluded, with its Support results retained.

Where Gorgias lands

We put ourselves through the same test.

Gorgias is one of the 18 vendors, on the identical blind rubric. Here is where we land, gap included, because a benchmark that hides its own is not worth reading.

#1 overall · #1 support · #4 shopping

Support quality

Support answer quality: #2 of 13, behind Sierra (78) (field median 65). Specific, grounded resolutions that close the ticket.

80%

Automation

80% shopping · 67% support across live stores: #3 of 13 (tied with Envive) on shopping automation, #4 of 13 on support.

Overall position

#1 support (68), #4 shopping (63), and #1 overall. Shopping answer quality: the highest in the field.

~18s

The gap: response speed

Shopping answers land at around 18s, in one block, slower than the fastest agents (Envive ~8s, Sierra ~10s). Support answers take about 14s. Shopping is the job where response time matters most.

Our Support Agent ranks #1 of 13 with answer quality 75/100 (#2 of 13, behind Sierra (78)). Our Shopping Assistant ranks #4 of 13, with answer quality 73, ahead of Envive (62) and one real gap: response speed. The answer and the product cards are right, they just arrive too slowly. That is the gap we are working on, and we are showing it here rather than hiding it.

Market learnings

Five things every leader evaluating an AI agent should know.

No vendor wins automation, quality, and speed at once.

Every vendor leads on one or two of the three and trails on the third. The first to close its gap will define the category, so treat any agent that claims all three today with skepticism.

The most common failure is not a wrong answer. It is a wall or a loop.

Authentication walls, repeated order-number demands, clarification loops, and handing everything to a human. Across the field, judges rarely caught made-up facts. They constantly caught agents that would not engage. These are guardrail problems, not model problems, and they are cheap to fix.

Configuration matters more than the model.

The same vendor scores well on one store and near zero on another. The result is won in onboarding and quality control, not in the model demo you were shown.

Response speed alone is a vanity metric, but slow is still slow.

The fastest agents are rarely the best, but speed still counts: it is a quarter of the shopping composite.

"AI chat" on the box is not always a real agent.

Some widgets are menus of canned replies. Some will not load for anyone but a human. Nearly a third of the widgets we detected produced no real conversation at all. You cannot tell from a demo, which is why a public, repeatable test is worth having.

Method

Real brands, cold sessions, free-typed questions only, no quick-reply shortcuts. Every conversation is captured, then scored blind by an LLM judge using binary, evidence-forced checks. The score is derived from the checks, never picked by the judge.

Quality rubric

Shopping /100: answer 30 · discovery 20 · recommendation 22 · rich elements 18 · close 10. Support /100: resolution 40 · accuracy 25 · actionability 20 · close 15. Spec: runner/eval-rubric.md.

Composite

Weighted per job. Shopping = 40% automation + 35% quality + 25% speed, because response speed drives conversion. Support = 50% automation + 40% quality + 10% speed, because containment is the job. Speed scores 100 at 3 seconds or under, 0 at 22 seconds or over.

Response speed

Mean time to the complete final answer, never stopped on a "let me check…" stall, never on quick-reply chips.

Neutrality

Every live, verified storefront a vendor runs counts, with no vendor-specific exclusions. A former Gorgias-only exclusion rule was removed in July 2026 for exactly this reason.

The real test for
ecommerce AI agents.