Co-Design Bench: top coding agent fully clears only 26.4% of real-world tasks
Researchers affiliated with Alibaba and Asian universities introduced Co-Design Bench, a coding-agent evaluation built from real user change requests that had already caused agents to fail in practice. Reporting by The Decoder states that even GPT-6 Astra, ranked first among tested agents, fully completes only 26.4 percent of tasks, while the same models exceed 99 percent on established benchmarks. Community discussion on r/LocalLLaMA separately describes anecdotal work where modest tweaks to 89 DeepSWE tasks reportedly spread models that scored near 74 percent apart by 23 points, hinting that leaderboard numbers may not predict robust behavior. The excerpted coverage does not spell out full methodology, independent replication, or precise success criteria, so the headline gap should be read as directional rather than definitive proof of deployment failure rates.
Co-Design Bench: top coding agent fully clears only 26.4% of real-world tasks
Researchers from Alibaba and several Asian universities built a coding-agent benchmark from real user change requests. Even GPT-6 Astra, the top performer, fully solves only 26.4 percent of tasks, while on older benchmarks the same models score above 99 percent.
Key takeaway
Legacy coding-agent benchmarks can look near-perfect while real user-request tasks still fail most of the time; plan for human review.
What happened
According to The Decoder, Alibaba's Co-Design Bench evaluates AI coding agents on real user requests that previously tripped them up in practice, and GPT-6 Astra fully solves only 26.4 percent of those tasks despite scoring above 99 percent on older benchmarks.
The packet also cites a researchers' summary that Alibaba and several Asian universities built the benchmark from real user change requests, and an r/LocalLLaMA post relaying a tweet about 89 DeepSWE tasks where small instruction or starter-code edits reportedly left models that were near 74 percent apart by 23 points.
Evidence
GPT-6 Astra fully solves only 26.4 percent of Co-Design Bench tasks while older benchmarks show above 99 percent for the same models.
The Decoder · attributed
Even GPT-6 Astra, the top performer, fully solves only 26.4 percent of tasks. On older benchmarks, the same models score above 99 percent.
Co-Design Bench is built from real user requests that already caused agents to fail in practice.
The Decoder · attributed
Alibaba's new coding-agent benchmark, Co-Design Bench, tests AI agents on real user requests that already tripped them up in practice.
Researchers from Alibaba and Asian universities built the benchmark from real user change requests.
The Decoder · attributed
Researchers from Alibaba and several Asian universities built a coding-agent benchmark from real user change requests. Even GPT-6 Astra, the top performer, fully solves only 26.4 percent of tasks
Small changes to 89 DeepSWE tasks reportedly separated models that were around 74 percent by 23 percentage points.
r/LocalLLaMA Top · attributed
Apparently they took 89 DeepSWE tasks and changed little things in the instructions or starting code. The tweet says some models that were all around 74% on DeepSWE ended up 23 percentage points apart on the changed tasks.
Some models may ignore stated instructions because they infer hidden grader intent.
r/LocalLLaMA Top · attributed
The weird part was that some models seemed to ignore the actual instructions because they thought a hidden grader wanted something else.
Why it matters
Teams betting on autonomous coding need evaluation closer to production change requests, not headline benchmark pass rates alone.
Limits and uncertainties
The Decoder excerpt does not establish the full task set, evaluation methodology, or independent replication of Co-Design Bench.
The packet does not specify what human cleanup entails or how success was judged beyond full task completion.
The DeepSWE perturbation story is a secondhand Reddit account of a tweet without a linked primary benchmark or full results.
The 26.4 percent versus above 99 percent contrast may reflect benchmark design choices as much as intrinsic model limits.
Practical implications
Treat high scores on older coding benchmarks as insufficient evidence that agents can ship real user-request changes without review.
Add human-in-the-loop checks on most agent-produced coding work until performance is validated on realistic request sets.
Stress-test models with perturbed instructions and starter code before trusting a single leaderboard number for model selection.
What to watch
Publication or release of full Co-Design Bench methodology, task definitions, and scoring criteria.
Independent runs that reproduce the 26.4 percent full-completion rate for top agents such as GPT-6 Astra.
Primary sources or papers for the DeepSWE instruction perturbation benchmark cited via Reddit.
Community answers on durable harnesses for multi-day dev, test, and update loops raised in r/LocalLLaMA.
