Every ecommerce AI agent,
tested the same way

How the ranking works

Each conversation is scored across three dimensions, and the composite is a weighted sum. The weights differ by lane, because post-sale conversations have more riding on resolution than pre-sale ones do.

Pre-sale = 40% automation + 35% quality + 25% speed. Post-sale = 50% automation + 40% quality + 10% speed. Speed scores 100 at ≤3s and 0 at ≥22s; a missing dimension is renormalised, not scored zero.

Why these weights, and not others

Automation is weighted heaviest in both lanes, because the main economic case for an AI agent is reducing human-agent load. A system that gives good answers but does not resolve conversations is not yet deployable at scale.

Answer quality carries the second weight, because poor answers damage customer relationships. A resolved ticket that leaves the customer with the wrong return window just creates a second, more expensive contact.

Speed is weighted lowest, because customers tolerate moderate latency when the response is accurate and resolves the issue. The floor is set at 22 seconds. Beyond that, the wait itself becomes the problem regardless of answer quality.

How we did it

Real stores. Every storefront was a live, mid-market ecommerce site carrying real inventory, 242 in all. No sandboxes, synthetic catalogues, or vendor-provided demonstrations.

Same questions. Every agent received the identical customer messages at the same time. No agent saw another agent's response before the scoring run completed.

Blind to the brand. The LLM judge evaluated responses with zero knowledge of which vendor produced them, against a fixed rubric of 26 binary checks, each one scored independently.

Code checks what code can. Factual checks like price accuracy, policy match, and SKU validity were validated programmatically against live storefront data, not by human recall.

Only rank what's real. A vendor needs at least 15 judged conversations in a job to earn a head-to-head rank, so no single storefront can carry it. Below that floor a vendor still appears but is not ranked.