EVIDENCE BEFORE RELEASE
Test the complete workflow, not only the model response.
An AI agent can produce a plausible answer and still fail the operation. Production evaluation must include the task, tools, data, permissions, workflow rules, exceptions, human handoff, records, and monitoring that surround the model.
The purpose is not to claim a universal benchmark. It is to build evidence that the specific agent behaves acceptably inside the specific process where it will be used.
That means evaluation should begin with the operating contract: what counts as success, what the agent is allowed to do, what it must refuse or escalate, what evidence must be created, and what conditions block release.
What to evaluate before production
Task success
Define what a completed task means, including required outputs, quality criteria, prohibited shortcuts, and the conditions that should produce no action.
Tool-use checks
Verify that the agent selects approved tools, passes correct parameters, respects permissions, confirms consequential actions, and handles tool failure.
Workflow adherence
Test required sequence, business rules, data validation, approvals, record updates, and handoffs instead of scoring only the final text.
Exception handling
Use missing data, conflicting instructions, unavailable systems, ambiguous requests, policy violations, and adversarial inputs to confirm safe behavior.
Human review
Confirm the reviewer receives enough context, can understand the proposed action, can correct it, and is not pressured into approving by default.
Regression testing
Keep representative cases so changes to prompts, models, tools, policies, or integrations can be compared against previously accepted behavior.
Build the evaluation set from the real workflow.
Start with the AI Workflow Audit and convert actual operating paths into test cases. A useful evaluation set should contain normal requests, boundary cases, known exceptions, missing information, unavailable integrations, conflicting instructions, unauthorized requests, and cases that should escalate to a person.
- Representative successful tasks from the normal workflow.
- Cases where required data is missing, stale, or contradictory.
- Requests that exceed the user’s or agent’s authority.
- Tool failures, timeouts, duplicate events, and unavailable dependencies.
- Ambiguous situations where the safe result is clarification or escalation.
- High-consequence actions that require explicit human approval.
Score behavior that matters to the operation.
| Dimension | What to verify |
|---|---|
| Correctness | The outcome matches the business rule and authoritative data. |
| Authority | The agent never exceeds its permitted tools, records, or transaction boundaries. |
| Sequence | Required validation, approval, and handoff steps occur in the correct order. |
| Recovery | Failures produce safe retries, escalation, rollback, or a manual path. |
| Evidence | The system records enough information to understand the request, action, approval, and result. |
| Human usability | A reviewer receives enough context to make a real decision rather than rubber-stamp the agent. |
Define release gates before the final demo.
Release criteria should be written before the team sees the final result. Otherwise, a convincing demonstration can quietly become the acceptance test.
A release gate can include required test coverage, zero tolerance for certain policy or authority violations, acceptable failure behavior, verified human handoff, operational monitoring, rollback readiness, and named ownership for incidents after launch.
Connect those gates to the AI Agent Governance Framework. Governance states the rules; evaluation supplies evidence that the specific system follows them.
Release in stages and monitor the same risks.
Begin with constrained permissions, limited users, reversible actions, and clear stop conditions. After release, monitor the same failure modes tested before release: task errors, exceptions, tool failures, policy breaches, escalation quality, cost, and unexpected changes in behavior.
A model, prompt, policy, integration, or upstream system can change after launch. Keep the accepted evaluation set and rerun relevant cases when those dependencies change. The broader AI Implementation Roadmap should assign ownership for that maintenance cycle.
AI Agent Evaluation FAQ
What should an AI agent evaluation test?
Test the complete task: inputs, reasoning boundaries, tools, permissions, workflow rules, approvals, records, exceptions, human handoff, and failure behavior.
Is model accuracy enough?
No. A correct answer can still produce the wrong business action, write to the wrong system, skip approval, or fail to preserve the required record.
When should regression tests run?
Run them when prompts, models, tools, policies, integrations, or other dependencies change enough to affect previously accepted behavior.
Who decides whether the agent is ready?
The organization should name the accountable business and technical owners and define the evidence they require before release.
Build a release gate your team can explain.
Read The Boring AI Letter for more practical AI systems guidance, explore the Nova Voice case study for a first-party implementation example, or discuss evaluation as part of a production build.