Case file · comparison
Agent Death Trap vs SWE-bench
SWE-bench is a serious benchmark and this page is not here to knock it. The two boards grade different things, and which one you should trust depends on what you are choosing a model for.
their board
What SWE-bench measures
SWE-bench measures software engineering: an agent gets a real GitHub issue from a public Python repository and has to produce a patch. The Verified subset is a human-filtered set of those issues.
Grading: A patch passes or fails the repository’s own unit tests. The headline number is the percent of issues resolved.
Training-data exposure: The issues and repositories are public, so they can appear in training data. The benchmark mitigates by date-splitting, but the underlying code is on GitHub.
this board
How Agent Death Trap differs
SWE-bench goes deep on one skill: writing code against a real codebase. Agent Death Trap goes wide: math, logic, single- and multi-step tool use, retrieval, long-context recall, instruction following, and state tracking, plus trap rooms for prompt injection, hallucination, sycophancy, and skill discipline.
SWE-bench has no safety axis. A model that patches every issue but leaks a secret when a tool result tells it to would still top the board. Here that leak costs real HP on the same scale as capability, though the heaviest hits are tool-use and skill-use discipline.
Room content is generated per seed from a seeded generator, so the exact problem a model sees was never published. SWE-bench tasks are fixed and public.
the honest answer
Which one to use
pick SWE-bench when
You are choosing a model for coding agents working in real repositories.
pick this trap when
You are choosing a general agent and need capability and safety in one comparable number.
They also compose. Run both and you know how a model codes and whether it stays honest under pressure.
Keep reading