The best AI coding agent fully finished only 26% of real user change requests in a new test

If your coding agent "almost" finishes everything, there's now a benchmark that agrees with you. A team from Alibaba and a handful of Asian universities assembled a test out of change requests from real users, and the top model fully completed only about one task in four, Jonathan Kemper reports for The Decoder.

The test

Co-Design Bench asks agents to rework existing web apps based on what users actually asked for, sometimes with images, audio, video or documents as input. The researchers favored requests where users had to rephrase or roll back changes, so the set is weighted toward work that already gave agents trouble. Jobs range from reading an amount off a receipt and fixing every total that depends on it to overhauling a whole project.

Grading is strict. An AI judge runs the finished app in a browser; a second judge reviews each pass, hunting for faked screens or hardcoded values, and only features that really work score.

The motivation: older benchmarks are saturated. Agents that can run and fix their own code score above 99% on them, so models are barely distinguishable.

The results

GPT-6 Astra led the ten models, covering roughly 70% of what each task asked for on average, yet getting the whole thing right only 26.4% of the time. It fully solved 41% of multimodal tasks and just under 18% of repository tasks. Claude Opus 5 and Qwen3.8-Max followed; Grok 4.6 came last.

Every model got worse as tasks mixed more kinds of change, across layout, design, interaction and content, which is how real projects usually look. Throwing compute at it didn't help: On small changes, GLM-5.2 burned nearly six times the tokens GPT-6 Astra did and still scored much lower.

The harness matters

The same model scored very differently depending on the harness that lets it read, write and run code. Qwen3.8-27B hit 75% or 56% on big overhauls depending on the environment, and which of two models won also flipped. The authors say leaderboards should report model and harness together, and companies should test them as a pair.

Limits apply: one fixed request per task, no follow-up conversation, and AI judges that can err. But it echoes METR's finding that maintainers accept AI code far less often than SWE-bench scores suggest.

The practical upshot: the agent does most of the work, and someone still has to finish it.

Scoreboard update: Humans +1. On real change requests, the best agent leaves a human cleanup job three times out of four.

Stay undead.