Testing AI Agents: The Pre-Launch Checklist

AI agents are tested by validating three things separately: task completion (did it achieve the goal), tool-use correctness (did it call the right tools with the right arguments), and failure recovery (what happens when a tool errors). Most teams only eyeball the first, which is why agents fail in production in ways demos never showed.

The checklist

• Golden tasks: 30–100 real multi-step tasks with verifiable end states, run on every prompt or model change
• Tool-call assertions: every expected call, argument shape, and ordering constraint checked, not just the final answer
• Failure injection: make each tool time out, error, and return garbage, the agent must recover or stop cleanly, never improvise state
• Loop guards: max-step and budget limits tested by giving the agent impossible tasks
• Permission boundaries: verify the agent cannot be prompted into tools or data outside its scope

The metric that matters

Track task success rate per version, like a test pass rate. When an agent change ships because 'it felt better in the demo', you're guessing, not engineering. Measured agents improve; eyeballed agents drift.