An AI agent benchmark is supposed to answer a simple question: if I hand this model tools and a goal, what happens? Not "can it answer trivia", but "can it plan, call the right tool with the right arguments, recover from errors, and stop when it is done".
That question turns out to be hard to score, which is why there are now a dozen agent benchmarks that all measure something slightly different. This guide covers what each family actually tests, what none of them test, and how Agent Death Trap fits in.
What an agent benchmark has to measure
Chat benchmarks grade one answer. An agent benchmark has to grade a loop: prompt, tool calls, observations, more tool calls, final answer. That loop can fail in ways a single answer never can:
- calling a tool with the wrong arguments, then trusting the garbage that comes back
- looping on a dead-end tool instead of backing out
- doing 90 percent of a multi-step chain and silently skipping the last step
- losing an instruction that was given 30 steps earlier
- getting talked into something by content it read mid-task
A useful agent benchmark has to see the whole trace, not just the final output.
The benchmarks people actually cite
- SWE-bench gives the agent a real GitHub issue and grades the patch against the repo's own tests. It is the standard for coding agents. It measures one domain, and its task set is public.
- AgentBench spans several environments (shell, database, web) and grades task success in each. Broad, but success-rate only.
- Terminal-Bench puts the agent in a terminal and scores completed tasks. Close to real ops work.
- GAIA asks assistant-style questions that are easy for humans and hard for models, with tool use required.
- τ-bench simulates a customer conversation with tools and a policy the agent must follow. It is one of the few that tests rule-keeping, not just success.
- WebArena and OSWorld test browser and desktop control in sandboxed environments.
All of these are worth reading. Side-by-side deep dives live on the comparison pages.
What almost none of them measure
Three gaps show up across the whole list.
Safety is scored elsewhere or not at all. A model that completes the task but leaks a credential along the way passes most of these benchmarks. Jailbreak resistance, sycophancy, and hallucination get their own papers with their own scores, and nothing ties them to the capability number. The result: a model can rank top-3 for capability while being trivially manipulable, and no single chart shows it.
Published task sets rot. Once tasks are public, they leak into training data. Scores drift up without the underlying skill improving.
Success rate hides the shape of failure. "62 percent solved" does not say whether the other 38 percent were near-misses or catastrophes. For an agent you plan to give real tools to, that difference is the whole story.
How Agent Death Trap scores an agent
Agent Death Trap runs every model through the same corridor of 14 rooms: 10 capability rooms (math, logic, tool use, retrieval, algorithms, multi-step tool chains, a tool maze, instruction following, state tracking, skill use) and 4 trap rooms (guardrail, hallucination, sycophancy, long-context recall). Every model starts at 100 HP. Each room grades the full trace with a deterministic verify() and returns one outcome from a fixed list. The outcome's damage comes from a public rubric. Hit 0 and the model dies, and the board records the room that killed it.
Three design choices matter:
- One number covers capability and safety. Trap damage is asymmetric: caving to pressure costs 25 HP, more than plainly failing a task. A capable but manipulable model cannot top the board.
- Rooms are generated per seed. The exact task text was never published, which blunts training-data contamination. A private holdout set of extra rooms is never published at all.
- Everything replays. Temperature 0, pinned model versions, seeded generation, capped steps and tokens, full traces logged. Same version, same seed, same result. The full contract is in the methodology.
What the current board says
On board v1.19.0 (40 models, 10 seeds each), the corridor is genuinely lethal: 7 models died. Four of the deaths happened in the tool maze, where the agent has to navigate tools that lie to it. The multi-step tool chain is the hardest room on the board: not one model of 40 cleared it perfectly. And the leader, gpt-5.6-sol, walked out with 93 of 100 HP, which is the point: nobody aces the corridor.
The live leaderboard ranks all 40. If you care about how specific vendors did, there are dedicated breakdowns for Claude and OpenAI's GPT line, and the safety benchmark guide covers the trap rooms in detail. Every model also has a post-mortem written from its recorded traces in the stories archive.
The board is free, no login, and the data is served as JSON and CSV from a public API if you want to run your own analysis.