Case file · comparisons
This trap, next to the other benchmarks.
SWE-bench, AgentBench, and Terminal-Bench are all serious benchmarks, and each measures something this one does not. The honest way to compare is to say what each one grades and when you should reach for it. That is what these pages do.
The short version: those boards measure capability. This board puts capability and safety on one HP scale in one run, so a model that does every task right but leaks a secret when tricked still loses.
versus
SWE-bench
SWE-bench measures software engineering: an agent gets a real GitHub issue from a public Python repository and has to produce a patch. The Verified subset is a human-filtered set of those issues.
full comparisonversus
AgentBench
AgentBench (THUDM, ICLR 2024) evaluates LLMs as agents across eight environments, including an operating system, a database, a knowledge graph, web shopping, and web browsing.
full comparisonversus
Terminal-Bench
Terminal-Bench measures how well an agent completes real tasks in a terminal: containerized environments, a task description, and a check that verifies the end state.
full comparisonside by side
At a glance
| Agent Death Trap | SWE-bench | AgentBench | Terminal-Bench | |
|---|---|---|---|---|
| What it measures | One corridor mixing capability rooms with trap rooms for safety and honesty | SWE-bench measures software engineering: an agent gets a real GitHub issue from a public Python repository and has to produce a patch. The Verified subset is a human-filtered set of those issues. | AgentBench (THUDM, ICLR 2024) evaluates LLMs as agents across eight environments, including an operating system, a database, a knowledge graph, web shopping, and web browsing. | Terminal-Bench measures how well an agent completes real tasks in a terminal: containerized environments, a task description, and a check that verifies the end state. |
| How it grades | Deterministic verify() returns one outcome from a fixed enum; damage comes from one public rubric | A patch passes or fails the repository’s own unit tests. The headline number is the percent of issues resolved. | Each environment reports its own success rate or reward; the paper aggregates per-environment scores. | Tasks pass or fail their verification script. Results are reported per harness and model pair, and the same model can score very differently depending on which harness wraps it. |
| Training-data exposure | Room content generated per seed; private holdout set never published | The issues and repositories are public, so they can appear in training data. The benchmark mitigates by date-splitting, but the underlying code is on GitHub. | The task sets are published in full on GitHub, so later models can have seen them during training. | Task definitions are public on GitHub. New versions rotate tasks to stay ahead of training data. |
The full scoring process, rubric, and limitations are on the methodology page. The board itself is the register.