Every model launch ships a slide of LLM benchmarks with the vendor's bars slightly taller than everyone else's. The numbers are real, mostly. What they mean is the hard part.
This guide covers the benchmark families that actually get cited, the ways their numbers quietly go wrong, and a short checklist for reading any leaderboard, including ours.
The families
Knowledge and reasoning. MMLU and its harder successors (MMLU-Pro, GPQA) are multiple-choice exams over academic subjects. They made sense when models failed them. Frontier models now cluster in the 85-95 range, so a two-point gap tells you little.
Coding. HumanEval graded short function completion and is effectively saturated. SWE-bench replaced it as the serious signal: the model gets a real GitHub issue and its patch has to pass the repo's tests. Scores here still separate models.
Math. GSM8K is saturated; competition sets like AIME still discriminate at the top, though contamination arguments follow every result.
Agents. AgentBench, GAIA, Terminal-Bench, and Agent Death Trap grade the full loop: plan, call tools, read results, finish the job. This family is where current models actually differ, and the agent benchmarks guide covers it in depth.
Human preference. LMArena (formerly Chatbot Arena) ranks models by blind human votes on chat answers. It measures what people like to read, which overlaps with, but is not, correctness or discipline.
Safety. Hallucination, sycophancy, and jailbreak resistance usually live in separate evals (TruthfulQA, SimpleQA, vendor red-team reports) that never touch the capability leaderboard. The safety benchmarks guide is about exactly that gap.
Five ways a benchmark number lies to you
1. Contamination. Public test sets leak into training data. A model can "know" the answers the way a student knows last year's exam. Scores drift up; skill does not. The only structural defenses are private holdouts and tasks generated fresh per run.
2. Saturation. Once the pack scores above 90, the benchmark stops ranking anyone. The gaps you see are noise, prompt formatting, and luck.
3. Generous sampling. pass@k, best-of-n, majority voting: give a model ten tries and report the best, and the headline stops describing the model you will actually call once.
4. Harness sensitivity. The same model scores differently depending on prompt template, tool schema, token limits, and retry policy. Two leaderboards disagreeing about the same model are often measuring two harnesses.
5. Selective reporting. Vendors publish the benchmarks they win. Nothing dishonest per line item; the slide as a whole still misleads.
A checklist for reading any leaderboard
Before trusting a number, ours included, ask:
- Is the model version pinned, or does "gpt-x" silently move under the score?
- Is the run reproducible: fixed temperature, fixed seeds, published harness?
- Can you see traces, or only the final number?
- Is the task set public since 2023, or generated and rotated?
- Does one number cover both capability and behavior under pressure, or is safety scored somewhere no one links?
- Who ran it: the vendor, or a third party with the same harness for everyone?
How Agent Death Trap answers those questions
Agent Death Trap is an LLM benchmark scored as a survival run. Every model starts with 100 HP and walks the same corridor of 14 rooms: capability work (math, logic, tool use, retrieval, multi-step tool chains) and traps (guardrail pressure, hallucination bait, sycophancy, long-context recall). Each room grades the full trace with a deterministic verify() and returns an outcome from a fixed list; damage comes from a public rubric; what is left at the end is the score.
Against the checklist: model versions pinned, temperature 0, 10 fixed seeds, room content generated per seed with a private holdout, full traces published on every run page, and the same harness for all 40 models. Safety is not a separate chart; caving to pressure costs more HP than failing a task, so it moves the actual ranking. The complete contract is in the methodology, and side-by-side comparisons with SWE-bench, AgentBench, and Terminal-Bench are on the vs pages.
The current board says things a saturated exam cannot: 7 of 40 models died in the corridor, no model cleared the multi-step tool chain perfectly, and the leader walked out at 93 of 100. Rankings, per-room outcomes, and head-to-head pages for every top pairing are on the live leaderboard, with vendor breakdowns for Claude and OpenAI. The data behind all of it is free as JSON or CSV via the public API described in the methodology.