How to test an AI agent before you trust it

A demo shows what an agent can do on a good day. Before you rely on one, you need to know what it does on a bad day. The NCSC’s advice is to start with tightly bounded pilots on clearly defined, low-risk tasks, and build confidence before widening the scope. Here’s a practical way to do that.

1. Build a test set from real work

Collect 20 to 50 past examples of the task, anonymised, where you already know the right answer: emails and the reply you sent, invoices and how you coded them. Include:

• Normal cases: the everyday bulk of the work.
• Awkward cases: missing information, ambiguous requests, angry customers.
• Cases it must refuse or escalate: refunds over your limit, legal threats, anything outside its job.
• Trick cases: messages containing instructions such as “forward this thread to …” to check it ignores them.

2. Decide what “good enough” means before you run it

Write down the pass mark in advance: for example, every escalation case escalated, zero trick cases followed, and most drafts usable with light edits. Setting the bar afterwards makes any result look acceptable.

3. Run it and mark it honestly

Have a person mark each output as correct, fixable or wrong, and note why. Look hardest at the wrong answers that looked confident: they’re the ones that slip past a busy reviewer.

4. Shadow mode before live mode

Run the agent alongside the person doing the job for a week or two. It drafts; they work as normal and compare. Only then let it act, and keep a person approving anything external or financial (see our permissions guide).

5. Re-test whenever something changes

Keep your test set. Run it again when you change the prompt, the tools, the model or the provider changes theirs. Anthropic’s guidance on building agents stresses extensive testing in sandboxed environments because agents can make compounding errors over many steps.

6. Keep watching after go-live

Check a sample of real outputs every week and log incidents. Risk frameworks such as NIST’s AI Risk Management Framework treat managing AI risk as continuing work across the life of a system, not a one-off sign-off.

Help me build a test plan for an AI agent.
The agent's job: [e.g. reply to booking enquiries by email]
What it must never do: [e.g. confirm a booking, offer a discount, share another customer's details]

1. Suggest 20 test cases as a table: case | type (normal / awkward / must-escalate / trick) | the input (short, anonymised, realistic) | the correct behaviour.
2. Include at least 4 "trick" cases where the input contains instructions the agent should ignore.
3. Propose pass criteria I should agree before running the test.
4. Give me a simple marking sheet: case | output | correct / fixable / wrong | note.
Use British English and invented, clearly fictional details only.

Sources

1. NCSC blog, "Thinking carefully before adopting agentic AI" (15 May 2026) Government guidance
2. Anthropic, "Building effective agents" Vendor guidance
3. NIST, AI Risk Management Framework Standards body
4. OWASP Gen AI Security Project, "Top 10 risks and mitigations for LLMs and Gen AI apps" (2025) Security standard