From SWE-Bench to FrontierCode: The New Agent Code Quality Era

TL;DR

• FrontierCode asks ‘would a maintainer merge this PR?’ using 3000+ rubrics built by 20+ open-source maintainers; slashing false positives by 81% vs SWE-Bench Pro.
• DeepSWE replaced SWE-Bench Pro in the AA Coding Agent Index after a ~32% grader error rate was uncovered; reshuffling rankings overnight.
• AA-AgentPerf introduced agents-per-megawatt as the first infrastructure-level benchmark for agentic inference; GB300 NVL72 delivered 20x more agents/MW than H200.

In June 2026, three benchmark releases landed within days of each other and collectively rewired how the industry measures coding agents . Cognition published FrontierCode, which asks not whether code passes tests. The question: would a maintainer merge the PR. Artificial Analysis swapped SWE-Bench Pro out of its Coding Agent Index for DeepSWE, after Datacurve’s audit revealed a ~32% grader error rate [7]. AA-AgentPerf introduced agents-per-megawatt, the first metric measuring how many concurrent coding agents your inference hardware sustains at defined SLOs [8]. The real story isn’t any single benchmark: correctness alone is no longer sufficient. Maintainability, contamination resistance, and throughput density are the new axes of evaluation.

Why SWE-Bench Pro Had to Be Replaced

SWE-Bench shaped coding agent evaluation for three years. In 2026, it collapsed under its own weight for three reasons that matter to anyone who used its scores to make model selection decisions.

First, the grader was wrong. Datacurve’s audit found SWE-Bench Pro’s automated grader carries a ~32% error rate: 8.5% false positives and 24% false negatives [7]. That error is enough to shuffle leaderboard rankings; a benchmark with a 32% failure rate produces scores you cannot trust as precision measurements.

Second, passing tests doesn’t mean mergeable code. METR recruited four maintainers from scikit-learn, Sphinx, and pytest to review 296 AI-generated PRs. The gap between automated grader pass rates and human maintainer merge decisions was roughly 24 percentage points, even after normalizing against human golden patches [11]. Nearly half the PRs that passed SWE-Bench would not survive a code review.

Third, the benchmark became gameable. Datacurve found Claude Opus recovering fixes directly from SWE-Bench Pro’s commit history. The model wasn’t solving the task; it was pattern-matching against the answer already present in repository metadata [5]. Any benchmark built from public GitHub data carries this risk.

FrontierCode: Grading Like a Tech Lead, Not a CI Pipeline

Cognition recruited over 20 open-source maintainers from 36 flagship repositories including Celery, Budibase, Mattermost, and uppy. Each spent 40+ hours building rubrics from their own codebases [1]. The benchmark evaluates across six maintainability dimensions: behavioral correctness, regression safety, mechanical cleanliness, test correctness, scope discipline, and code quality. Blocker criteria are built into 3000+ structured rubrics; fail any single blocker and the solution scores zero regardless of passing tests [1][3].

Consider the jsonschema C++ LOG_WARNING task. Claude Opus 4.8’s patch passed all tests. The behavior was identical. But the model mixed LOG_WARNING() and std::cerr in multi-line warning messages, creating structural fragility that any refactoring would break. A single blocker criterion zeroed the entire score [1]. This is the call a senior engineer makes daily that automated graders never catch.

FrontierCode achieves 81% fewer false positives than SWE-Bench Pro through four grading techniques: reverse-classical tests that validate the negative space, code scope checks ensuring the patch touches only what it should, adaptive classical grading calibrated per-task by the maintainer, and mutagent, a mutation-based adversarial grader [1]. Prompts are about one-third the length of SWE-Bench Pro equivalents; agents must infer maintainer intent rather than follow detailed instructions.

How FrontierCode Reshuffled the Model Rankings

FrontierCode’s three-tier structure reveals what a single-score benchmark hides. On Diamond, the 50 hardest tasks, Claude Opus 4.8 leads at 13.4% while GPT-5.5 scores 6.3% and Kimi K2.6, the best open-source model, reaches just 3.8% [1][10]. No model reaches 15% on the hardest coding tasks ever publicly benchmarked: the ceiling is real, and the gap between frontier models and everything else has never been wider.

On Main (100 tasks), Opus 4.8 rises to 34.3% [1]. On Extended (150 tasks), it hits 51.8%; the easiest third nears saturation [1]. Historical data shows rapid learning: Opus nearly doubled its Extended score from 41% to 74% in four months during late 2025 runs [3].

Token efficiency tells a different story. GPT-5.5 uses up to 4x fewer tokens than Opus 4.8 on comparable results [1]. On a cost-per-task basis, the lower Diamond score may represent a better tradeoff if your workload runs thousands of agent invocations daily. The open-source gap is the widest on any coding benchmark: Kimi K2.6’s 3.8% Diamond versus Opus 4.8’s 13.4% is a 3.5x delta [10].

graph TD
  subgraph "Diamond Tier [1]"
    A["Opus 4.8: 13.4%"] --> D["All below 15%"]
    B["GPT-5.5: 6.3%"] --> D
    C["Kimi K2.6: 3.8%"] --> D
  end
  subgraph "Token Efficiency [1]"
    E["GPT-5.5 uses 4x fewer tokens"]
  end
  subgraph "History [3]"
    F["Opus: 41% → 74% in 4mo"]
  end

DeepSWE and the Artificial Analysis Index Swap

On June 12, 2026, Artificial Analysis dropped SWE-Bench Pro from its Coding Agent Index and replaced it with Datacurve’s DeepSWE, citing gameability via commit-history recovery [4][5]. The swap reshuffled the rankings completely.

DeepSWE consists of 113 tasks across 91 repositories in five languages: TypeScript, Go, Python, JavaScript, and Rust [5][6]. Every task is written from scratch, making contamination impossible. Behavioral verifiers achieve a 0.3% false positive rate versus SWE-Bench Pro’s 32% total error [7]. Solutions require roughly ~668 lines of code versus SWE-Bench Pro’s ~120 lines; longer-horizon, more realistic software engineering tasks [6].

Codex with GPT-5.5 (xhigh) rose from 65 to 76, overtaking Claude Code with Opus 4.8 (max) at 73 [4][5]. Claude Fable 5 entered at 77 but was later revoked under export controls [12]. SWE-Bench Pro’s error rate systematically flattered some combinations and penalized others. DeepSWE’s behavioral verifiers exposed which could genuinely solve engineering tasks versus which had learned to game blind spots.

AA-AgentPerf: Agents per Megawatt as the New Infrastructure Metric

Once you’ve found a model that writes mergeable, contamination-free code, a harder question remains: how many concurrent agents can your hardware sustain, and at what power cost? AA-AgentPerf is the first benchmark built specifically for agentic inference workloads [8][9].

It uses real coding-agent trajectories up to 200 turns, sequences exceeding 100K tokens, across 12+ languages. The mean agent request is ~27K tokens, far beyond standard LLM serving benchmarks [9]. Agent workloads have distinctly different patterns: long context windows, intermittent tool calls, and KV cache behavior unlike chatbot serving.

Same model (DeepSeek V4 Pro), same trajectories, same SLO: GB300 NVL72 delivers 61,354 agents per megawatt versus H200’s 2,594, a roughly 20x improvement [8][9]. Rack-scale disaggregated inference on GB300 is ~3x more power-efficient than single-node B300 [8]. Production optimizations (KV cache reuse, speculative decoding) are permitted; the benchmark proxies real deployment economics, not raw FLOPs.

Build a Three-Layer Eval That Catches What SWE-Bench Misses

Together, these benchmarks define an evaluation stack: Layer 1 (Code Quality) asks whether a maintainer would merge the PR [1]. Layer 2 (Benchmark Integrity) demands contamination-resistant, behavior-based verification [5][7]. Layer 3 (Infrastructure Efficiency) measures agents per megawatt on real trajectories [8][9].

If your internal eval only runs unit tests, you’re at the SWE-Bench Pro level: 81% higher false positives than FrontierCode (and blind to structural issues maintainers catch daily) [1]. A model that writes mergeable code (Layer 1) and passes contamination-free verification (Layer 2) may still cost too much per invocation (Layer 3). The full picture requires all three.

This mirrors swyx’s three eras: HumanEval (2021) tested autocomplete snippets, SWE-Bench (2023) tested single-file patches, and FrontierCode (2026) tests production-grade code [3]. The arc from ‘can the model write a correct function?’ to ‘can the model produce mergeable, contamination-free code at viable throughput?’ is the arc of the entire field.

Choose Your Benchmark by What It Proves, Not What It Scores

DeepSWE’s design bets: contamination-free tasks eliminate an entire class of evaluation failure, behavioral verifiers with 0.3% false positives make scores trustworthy, and multi-language coverage prevents overfitting to Python idioms [5][6].

What remains unresolved: SWE-Bench Pro’s 32% error rate means years of published leaderboard data is unreliable [7]. No new benchmark quality retroactively corrects old decisions. Agent Arena offers a complementary approach with real-world sessions but introduces selection biases from its user population [2]. We don’t yet know how contamination resistance, behavioral verification, maintainability scoring, and throughput measurement trade off against each other in production.

Practical Implications for Agent Engineering Teams

Stop using SWE-Bench scores as your primary model selection signal: the numbers are damning. Thirty-two percent. That is the grader error rate [7]. Add the ~24pp maintainer rejection gap [11]. Neither is theoretical. Re-evaluate against DeepSWE or FrontierCode before your next infrastructure refresh.

Build evaluation pipelines that assess more than correctness. Add scope discipline (does the patch touch only what it should?) and code quality (would a senior engineer merge this?). Two extra dimensions already place you above the SWE-Bench Pro standard [1].

Treat harness quality as a first-class variable. Claude Code with Opus 4.8 scores 73 on the AA Index; the same model through a different harness scores differently [4][5]. Benchmark your full model times harness combination.

Factor infrastructure density into model economics. GPT-5.5 uses 4x fewer tokens than Opus 4.8 [1]; GB300 delivers 20x more agents per megawatt than H200 [8]. Combined, these create a potential 50x to 80x cost gap between deployment configurations. Measure agents per megawatt on your actual workload, not tokens per second on synthetic benchmarks.

Practical Takeaways

1. Re-evaluate your current model selection. If it was based on SWE-Bench Pro scores from before June 2026, cross-reference against DeepSWE or FrontierCode before your next infrastructure refresh. The benchmark had a 32% error rate and an unknown contamination ceiling.
2. Add at least two evaluation dimensions beyond correctness: scope discipline and code quality. Two extra dimensions already place you above the SWE-Bench Pro standard, which measured nothing beyond test pass/fail.
3. Benchmark your full model times harness combination. Claude Code, Codex, and direct API access produce different results from the same underlying model. Your harness is part of your architecture.
4. Replace tokens-per-second with agents-per-megawatt in capacity planning. Agent workloads have distinctly different compute patterns than chatbot serving; plan for long-context trajectories, not token benchmarks.

Conclusion

June 2026 was a reset, not a recap. Swap your reference data from the old leaderboard to the new axes: mergeability, contamination resistance, and throughput cost. If your current picks hold up, you bought well; if they slide, you are making decisions on stale numbers. The question for 2027: would a maintainer merge it, on clean data, at a viable cost per agent?

Frequently Asked Questions

Should we stop using SWE-Bench scores entirely?

Yes, for model selection decisions after June 2026. The 32% grader error [7] and contamination from commit-history recovery [5] make pre-June-2026 scores unreliable as primary inputs. Cross-reference against DeepSWE or FrontierCode. Historical SWE-Bench scores still carry value for trend analysis if you treat them as noisy proxies, not precision measurements.

How do we build internal evals that capture what FrontierCode measures?

Add scope discipline and code quality checks. A 5-point rubric plus a scope check already beats SWE-Bench Pro; see the comparison table above.

Does the GB300’s 20x advantage over H200 translate to my cloud deployment?

Not directly. The 20x figure comes from controlled benchmarks on specific hardware [8]. Your improvement depends on your model, agent architecture , request distribution, and cloud instance types. What the benchmark gives you is a new vocabulary: measure agents per megawatt on your actual workload. If your inference provider can’t provide this metric, treat it as a red flag.

Why did GPT-5.5 overtake Claude on DeepSWE but trail on FrontierCode?

These benchmarks measure different things. DeepSWE tests task-solving from open-ended prompts; FrontierCode adds maintainability grading on top. The jsonschema C++ case where Opus 4.8 was zeroed for mixing LOG_WARNING() and std::cerr despite correct behavior [1] is exactly the structural detail FrontierCode catches. A model can be better at solving tasks and still produce less mergeable code.

What’s the minimum eval setup for a team with 5 to 10 engineers?

Run your model against the public DeepSWE subset for a contamination-free baseline [5]. Add scope discipline to CI: a short script flagging agent PRs touching files outside expected scope catches most violations. Do not skip throughput. Agents per megawatt matters more than tokens per second for real workloads, and the gap between deployment configurations can reach 50x to 80x depending on your hardware. Track tokens per task invocation; GPT-5.5’s 4x token efficiency over Opus 4.8 [1] means the cheaper model sometimes wins on total cost. We don’t have clean cross-benchmark production data on this yet. Test with your own task distribution.

Sources

Image Credits

• Cover photo: Image generated with flux-pro-1.1 (Agents’ Codex AI illustration)
• Figure 1: Image generated with flux-pro-1.1 (Agents’ Codex AI illustration)
• Figure 2: Image generated with flux-pro-1.1 (Agents’ Codex AI illustration)