Test & evaluate agents
An agent that performs well on one ideal prompt is not ready for a workflow. Test the behavior you expect under normal and failure conditions.
Build a small evaluation set
Include at least:
- a normal task the agent should complete
- an ambiguous task that should trigger clarification or careful assumptions
- a task with missing evidence
- a request that conflicts with the agent’s constraints
- a task requiring a tool the agent does not have
- a task that should be refused or escalated because of permission/approval boundaries
Evaluate more than writing quality
Check whether the agent:
- follows the durable role
- preserves required constraints
- uses tools when evidence is needed
- distinguishes facts from assumptions
- does not claim validators or external actions ran when they did not
- stays inside Project/Knowledge scope
- handles unavailable capabilities clearly
For tool-backed agents, test tool failures too. The agent should not convert a failed search, repository lookup, validator, or external action into a confident success claim.
Permission tests
Use a test account/context with narrower permissions to verify the agent does not depend on accidental administrator access.
Regression testing
After changing agent instructions, tools, model eligibility, or Knowledge sources, rerun the same evaluation set. A change that improves one scenario can break another.
Automation readiness
Before attaching an agent to a schedule, watcher, or multi-step workflow, verify that it behaves safely without constant conversational correction.
Treat an important agent like a small software component: define expected behavior, test it, change one thing at a time, and rerun the checks.
