A practical checklist for evaluating AI agents before production

An agent demo answers one question: can the model complete a happy-path task once? A production evaluation has to answer a harder one: can the whole system complete useful work repeatedly, stay within its authority, recover safely, and make its result easy to check?

That distinction is worth making before choosing a model. Most agent failures are not eloquent wrong answers. They are state changes made with the wrong account, an incomplete handoff presented as done, a retry loop that burns budget, or an instruction in retrieved content treated as permission.

Use this checklist to turn a promising prototype into an evaluation plan.

1. Define completion before testing capability

Write down the postcondition that proves the task is complete. “The agent says the invoice was sent” is not a postcondition. “The invoicing system shows invoice INV-1042 delivered to the approved customer and no duplicate exists” is.

For every workflow, define:

• the required final state;
• the records that prove it;
• the allowed side effects;
• the side effects that require confirmation; and
• how the workflow can be reversed if it goes wrong.

This prevents a common evaluation mistake: scoring a polished final sentence as success while the underlying transaction is still only drafted or staged. The tool-calling failure guide shows why the last verification step matters as much as the tool call itself.

2. Test a trace, not just an answer

An agent’s output is a sequence: it chooses tools, supplies arguments, reads observations, retains state, retries, and eventually reports an outcome. Grade that sequence.

Capture at least the prompt, tool definitions, every call and response, retry count, final answer, and the observed final state. Then make tests fail when the path is unsafe even if the final state happened to be correct. For example, an agent should not earn a clean success after sending a live write when the task required a dry run first.

Agent Death Trap follows this principle: its stations verify recorded behavior and assign a fixed outcome, rather than trusting a self-reported claim of success. Read the methodology for the exact contract.

3. Make the ordinary path representative

Build a small task set from real work, not only tasks invented because they are easy to score. Include the normal inputs your team sees: incomplete requests, long records, closely named accounts, stale documents, and systems that return partial results.

Keep a separate holdout set for release decisions. If engineers repeatedly inspect and tune against the same cases, the test becomes a memory check. A useful split is:

Score each set with the same evaluator. Do not let a model see the answer key through a tool description, a fixture name, or a prompt template.

4. Add adversarial but realistic cases

Production inputs are not neutral. A support ticket can include a request to ignore policy. A search result can recommend an obsolete API. A tool can return a plausible decoy ID. A customer can confidently assert a false premise.

Test whether the agent:

• treats retrieved text as data, not as authority;
• distinguishes instructions from untrusted documents;
• asks for clarification when the target is ambiguous;
• preserves opaque IDs instead of replacing them with a similar-looking value; and
• refuses or escalates actions outside its scope.

These cases are not edge-case theater. They reveal whether a capable model can be safely connected to real systems. The LLM safety benchmark guide explains why safety performance needs to be considered alongside task completion.

5. Measure recovery, cost, and variance

One successful run is not a reliability result. Run each case more than once under fixed conditions, then record completion rate, unsafe-action rate, latency, tool calls, tokens, and cost. Report the spread, not only the average.

Also inspect recovery quality. A useful retry changes a hypothesis: it reads the schema, chooses another record, or asks a targeted question. Repeating an identical request after an identical error is not recovery. It is a loop.

For comparisons, keep the workload, provider settings, tool environment, and timeout budget constant. Otherwise a lower price may merely mean the system stopped earlier or skipped verification. The cost guide covers the difference between a cheap call and a cheap agent workflow.

6. Decide what code should own

The model should make judgments where language and ambiguity matter. Application code should own deterministic constraints: schema validation, authorization, idempotency, rate limits, state persistence, and postcondition checks.

This boundary improves both reliability and evaluation. Instead of asking the model to recite an account token correctly in prose, keep the token in typed state and let the model select from validated options. Instead of hoping it remembers to use dry-run mode, make live mode require an explicit approved transition.

A release gate worth using

Before enabling a new agent version, require all of the following:

1. It passes the representative holdout tasks at the agreed threshold.
2. It has no unapproved high-impact side effects in safety tests.
3. Its failure modes are observable and recoverable.
4. Its cost and latency fit the operational budget, including retries.
5. A human can audit what happened from the trace.

The exact threshold depends on the task. A calendar-drafting agent can tolerate more uncertainty than an agent that changes access controls. The important move is making that risk decision explicit before the model is connected to production authority.