A refusal model is useful, but it is not an agent safety system. Once an agent can search, read customer records, send messages, or make changes through tools, safety is determined by the full stack: what the model can see, what it is allowed to do, which tool calls are valid, and how a human can intervene.
The goal is not to make an agent refuse everything uncertain. It is to let it complete safe, useful work while making harmful or unauthorized actions difficult to initiate and easy to catch.
Start with authority, not with prompts
For each agent, describe its authority in concrete verbs. “Can help with support” is too broad. “Can look up an order, draft a reply, and issue a refund below a stated limit after confirmation” is testable.
Then enforce the boundary outside the model:
- give the agent read-only credentials by default;
- expose narrow tools instead of a general administrator API;
- restrict each tool to the current tenant, user, and workflow;
- require a separate approval token for sensitive writes; and
- expire elevated access quickly.
Prompt instructions remain valuable for setting behavior, but they are not access control. A model can misunderstand, be manipulated by untrusted content, or simply make a mistaken tool call. The tool layer must still reject an action that the policy does not permit.
Separate trusted instructions from untrusted content
An agent may process emails, web pages, tickets, PDFs, and tool responses that contain imperative language. Those sources can say “ignore the previous rules,” “send this export,” or “use this secret.” That is content to analyze, not an instruction to execute.
Make that distinction visible in the agent’s context and architecture. Label retrieved material as untrusted. Do not concatenate it into the same instruction channel as the workflow policy. When an input asks for a new action, require the agent to map it back to the user’s stated goal and its allowed authority.
This is especially important for retrieval and browser agents. The relevant question is not whether the model recognizes a theatrical jailbreak phrase; it is whether an untrusted page can cause a real side effect. The benchmark’s safety guide covers this class of failure in more detail.
Put confirmation where consequences begin
Not every action needs a human click. Too many confirmations train users to approve without reading. Put confirmation gates at actions that are irreversible, costly, external, or difficult to undo:
| Action | Appropriate control |
|---|---|
| Search a knowledge base | Allow automatically, log it |
| Draft an email | Allow automatically, show draft before send |
| Send a customer message | Confirm recipient, content, and channel |
| Change permissions or secrets | Require elevated approval and a scoped token |
| Move money or delete data | Require strong confirmation and an independent check |
Show the human the exact target and consequence. “Approve action?” is weak. “Send this refund of $85 to order #1042?” gives them something meaningful to validate.
Validate every tool call deterministically
The model should not be responsible for being its own final validator. Every write-capable tool should validate its arguments, authorization, expected state, and idempotency key before acting.
Good tool errors are precise enough to support recovery: identify the missing field, invalid enum, stale record, or policy violation. Do not leak secrets in error text, and do not silently coerce ambiguous arguments into a live action.
After a call, check the postcondition against the system of record. A message is not sent because the agent said it sent it; it is sent when the provider reports the expected delivery state. This is the same discipline that prevents partial tool chains from being mistaken for success in tool-calling workflows.
Design graceful failure paths
Safety mechanisms should leave the user with a useful next step. When the agent lacks permission or sees an ambiguous target, it should state what is missing, preserve its draft or plan when appropriate, and request the narrowest clarification or approval needed.
Avoid unbounded retries. If a tool response and request have not changed, another attempt is unlikely to help. Cap attempts, record the failure, and escalate when the remaining uncertainty is outside the agent’s authority.
Keep an audit trail that answers five questions
For each consequential run, you should be able to answer:
- What did the user ask for?
- What information and instructions did the agent receive?
- Which tools did it call with what validated arguments?
- What state changed, and how was that change verified?
- Who approved the action when approval was required?
Logs need appropriate privacy controls, but an absent trace makes incident review and benchmark-driven improvement impossible. Full trajectories are also how you distinguish a sound outcome from a fluent explanation of an action that never happened.
Test the guardrails independently
Do not infer safety from a successful task-completion score. Test blocked actions, prompt injection in retrieved text, misleading tool descriptions, wrong-recipient requests, stale permissions, and partial failures. Make sure the tests exercise the policy enforcement code, not just a sentence in the system prompt.
Agent Death Trap deliberately makes safety outcomes costly because capability without resistance is not a production-ready result. Use a benchmark as one input, then test the particular authority and data paths your deployment will expose. A safer model helps; a well-designed permission boundary helps every model.