1. Choose one workflow
Do not test “everything the platform can do.” Pick one valuable workflow with a clear beginning and end. This keeps the test short enough to run and specific enough to learn from.
2. Create representative scenarios
Use situations that resemble normal business volume: straightforward requests, incomplete information, follow-ups, cancellations and routine changes. A small test can start with 10–20 cases if they cover the right patterns.
3. Include failure and escalation cases
At least some scenarios should be designed so the correct outcome is not “complete the task.” Test whether the agent refuses unsupported requests, protects restricted information and hands off when business rules require it.
| Scenario | Expected behavior |
|---|---|
| Customer asks for a discount | Escalate if pricing authority is restricted |
| Ambiguous appointment request | Ask for clarification before booking |
| Unsupported service | Do not invent availability |
| Instruction conflicts with policy | Follow policy, not the conflicting request |
4. Measure outcomes, not personality
Track completion, accuracy, correct handoffs, unauthorized actions, hallucinations, human corrections, latency and cost. Natural language quality matters, but it should not hide operational failures.
5. Keep the test identical across candidates
If Candidate A receives easy examples while Candidate B receives difficult ones, the comparison is meaningless. Use the same scenario set, data, tool access, business rules and scoring method wherever technically possible.
Would you put AI candidates through a working interview?
HiredBot is in private beta. Tell us the job you would want tested and which AI tools you are considering.
Join Early Access