All stories

Cheapest reliable LLM for AI agents: what the benchmark says

2026-08-02 · benchmark v1.19.0

Cheapest reliable LLM for AI agents: what the benchmark says

On the current Agent Death Trap board, gpt-5.6-luna is the cheapest model that still looks reliable enough for general agent work. It finishes with 86 HP at $0.1002 per run, survives the corridor, and is one of only four models with a perfect aggregate result in the tool maze.

That answer needs a definition. "Reliable" is not a property a benchmark can certify for every production system. For this comparison, it means:

  1. the model survived the full 14-room corridor,
  2. its average final score is at least 70 HP across 10 fixed seeds, and
  3. cost was recorded for the same benchmark run.

Seventy HP is a screening threshold, not a law of nature. It removes models whose low price comes with repeated safety failures, broken multi-step execution, or death in the corridor. Raise the threshold to 85 HP and the winner is still gpt-5.6-luna.

The affordable shortlist

Board v1.19.0 contains 44 models. These are the six cheapest that pass the 70-HP screen.

Model Final HP Cost / run HP / $ Latency
gpt-5.6-luna 86 ±7.9 $0.1002 858 122.0s
gpt-5-mini 75 ±5.3 $0.1005 746 514.7s
gemini-3.5-flash-lite 70 ±16.8 $0.1039 674 216.7s
gpt-5.3-chat 81 ±6.9 $0.1380 587 93.4s
gpt-5.6-terra 89 ±7.2 $0.1667 534 85.6s
gpt-5.2 88 ±3.0 $0.2025 435 147.3s

The cost figures are observed per-run charges from this harness, not universal API quotes. Your prompt length, output length, cache hit rate, provider, region, and retry policy will change the invoice. The useful comparison is what each model accomplished for the cost recorded on the same workload.

Why luna is the default pick

The most revealing comparison is not luna against a flagship. It is luna against gpt-5-mini. Their benchmark runs differ in price by just $0.0003, but luna finishes 11 HP higher and about four times faster. GPT-5-mini is more consistent across seeds, yet it loses more ground in the guardrail and tool maze rooms.

Luna's tool record is unusually strong for the price. It clears the grounded tool-use room, gets a partial result in the underdocumented multi-step tool chain, and a perfect aggregate outcome in the tool maze. The safety record is also clean enough for a default agent: resisted at the guardrail, resisted under sycophancy pressure, unsupported rather than hallucinated in the document room, and recalled the long-context instruction.

The caveat is variance. Luna's ±7.9 spread is wider than gpt-5.2's ±3.0 or gpt-5-mini's ±5.3. If one bad run is expensive, the cheapest reliable architecture may be luna with validation and escalation rather than luna alone.

The cheapest API call is not the cheapest agent

Several models cost far less per run:

  • deepseek-v4-flash costs $0.0100 but finishes at 34 HP after hallucinating and caving under sycophancy pressure.
  • grok-4-fast costs $0.0162, finishes at 7 HP, and dies in the tool maze.
  • Qwen3-32B costs $0.0163, finishes at 4 HP, and also dies in the tool maze.

Those can be good models for constrained subtasks. They are not cheaper if the workflow needs retries, human review, rollback, or a second model to verify every action. Agent cost is:

model calls + retries + validation + recovery + expected cost of bad actions

The last term is why raw token pricing is a poor buying guide for autonomous systems.

When to spend more

Choose gpt-5.6-terra when latency matters. It reaches 89 HP and finishes in 85.6 seconds on this run. At $0.1667, it buys three more HP and roughly 36 seconds over luna. It is the stronger option for an interactive agent where waiting is part of the product experience.

Choose gpt-5.2 when variance matters. It finishes at 88 ±3.0 HP. The extra ten cents per run buys a much tighter distribution than luna, which may be worth more than average score in a workflow that cannot tolerate occasional weak runs.

Choose a premium model when validation is impossible. If the agent must make a high-impact judgment that cannot be checked mechanically, this cost screen is too permissive. Start with the highest-quality models, then measure whether a smaller model can safely take over specific steps.

A practical routing policy

The best budget architecture is usually a ladder:

  1. Run a low-cost model such as gpt-5.6-luna for routine, reversible tool work.
  2. Validate tool arguments and final state in code wherever possible.
  3. Escalate ambiguous, high-impact, or repeatedly failing cases to a more consistent model.
  4. Log full traces and price retries, not just first attempts.

That policy uses the cheap model where it is genuinely cheap and prevents one weak run from becoming an expensive incident. See why agents fail at tool calling for the failure patterns the validator should catch, and use the live comparison pages to test different quality, latency, and cost cutoffs.

Current rankings on the leaderboard, scoring details in the methodology.

Get told when a new model runs

One email when a new benchmark version lands or a new model walks the corridor. Nothing else.