Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
Paper recorded by Signals 4 on 2026-09-15 in cs.AI. Abstract reproduced from arXiv; link to the original below.
Published 2026-09-15 on arXiv · recorded by Signals 4 on 2026-09-16
Category: cs.AI · 人工智能 · first seen 2026-09-16
Abstract
Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models. On Verified, the leading two entries each resolve 396 of 500 instances. The top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes
Read on arXiv →
Cite this page: Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead: the #57 most recent of 300 cs.AI papers we have recorded (as of 2026-09-15). Source: Signals 4 (Signals API) — https://data.jiangzhang.ca/signals4/t/papers/coding-agents-have-converged-why-the-swe-bench-leaderboard-can-no-longer-order-i.html
Free to quote with attribution to “Signals 4 (Signals API)”. Machine-readable:
papers.json
Get 4 AI signals a day by email — free.
Get 4 AI signals a day by email — free