Pre-sale weighting
The rubric
Measure your own AI agent
Download the rubric we used to score every vendor, then grade your own AI agent.
Download the rubric we used to score every vendor, then grade your own AI agent.
Your download is ready.
Twenty-six pass/fail checks, scored out of 100 per lane. A blind judge answers each one yes or no and must quote the transcript to pass. Code does the scoring, not the judge. Two rubrics, one per job.
Download the rubric we used to score every vendor, then grade your own AI agent.
Your download is ready.
Pre-sale weighting
Post-sale weighting
| Check | Passes when | Points |
|---|---|---|
| Did it answer you | 30 | |
| Answered the question | Direct, substantive reply to each question. A generic greeting or boilerplate fails. | 14 |
| Didn't contradict itself | No contradictions across turns, no invented policies or specs. | 9 |
| Left nothing unanswered | No question left unanswered or answered with something unrelated. | 7 |
| Did it work out what you needed | 20 | |
| Asked before recommending | Asks a relevant clarifying question on a vague opener before recommending. | 8 |
| Built on your answers | Uses the shopper's answers to refine across turns, builds a profile. | 7 |
| Didn't dump a product list | Does not dump a generic best-seller list before understanding needs. | 5 |
| Did it recommend well | 22 | |
| Named a specific product | Recommends at least one specific, named product. | 9 |
| Explained why it fits | Ties the pick to the shopper's stated constraints with real reasoning. | 8 |
| Suited the store and the ask | Recommendations fit the store's catalog and the request. | 5 |
| Did it show you the product | 18 | |
| Code-checked, not judged. These four pass only if the link, price, review, or comparison actually appears in the transcript. Code detects each one, so a check can't be credited for something that isn't there. | ||
| Gave a link to open | An openable product link or card. | 7 |
| Showed a price | A concrete price attached to a recommended product. | 6 |
| Backed it with reviews | Review counts or ratings support the recommendation. | 3 |
| Offered a comparison | Multiple distinct options laid out to compare. | 2 |
| Did it help you buy | 10 | |
| Gave a next step | Ends with a concrete next step. | 5 |
| Completed the purchase | Completes add-to-cart or checkout when explicitly asked. | 3 |
| Finished cleanly | Closes cleanly, no dangling question or mid-thought stop. | 2 |
| Check | Passes when | Points |
|---|---|---|
| Did it solve the problem | 40 | |
| All three fail if it sent you away. If the agent pushed you to "email us" or "call us" on a question it could have answered in the chat, this whole group scores zero, whatever else the reply said. Sending you elsewhere is not solving it here. | ||
| Answered it in the chat | Primary ask gets a complete, store-specific answer in-channel. A generic pointer fails. | 18 |
| Ended the right way | Outcome-correct ending: resolved, or a justified handover done well. An unjustified bail fails. | 12 |
| Didn't send you away | Does not push out of channel ("email us") when the question was answerable in chat. | 10 |
| Was it accurate | 25 | |
| Gave real facts | At least two concrete, store-specific facts. One vague reassurance fails. | 13 |
| Sounded like this store | Claims read as grounded in this store, not industry generalities. | 7 |
| Didn't contradict itself | No self-contradiction across turns. | 5 |
| Could you act on it | 20 | |
| Told you what to do | Customer leaves with steps they can execute now, in a cold session. "Reach out if…" alone fails. | 12 |
| Answered all of it | Covers the actual ask, not a fragment. | 8 |
| Did it close properly | 15 | |
| Said what happens next | Sets expectations on who acts and when. Vague "soon" fails. | 8 |
| Finished cleanly | Clean, complete close. | 7 |
This rubric is the quality score. A vendor's lane ranking also weighs automation rate and speed. Shopping: 40% automation, 35% quality, 25% speed. Support: 50%, 40%, 10%.
The rubric
Download the rubric we used to score every vendor, then grade your own AI agent.
Download the rubric we used to score every vendor, then grade your own AI agent.
Your download is ready.