No AI agent scores above 12% on SemiAnalysis' eight-hour real-work benchmark

If you've been told AI agents are ready to do knowledge work, here's a cold shower. SemiAnalysis has released GDPx, a benchmark that drops agents into simulated jobs instead of quizzing them, and no model clears 12%, Manuel Uth reports for The Decoder.

A test that looks like a workday

There are 206 tasks across eleven fields, among them law, cybersecurity, biology, chip design and quant trading. Agents spend up to eight hours in each environment: running professional software like Siemens NX, searching data rooms that hold thousands of files, or breaking into live corporate systems. SemiAnalysis pitches it against OpenAI's GDPval, which it says hands models all the context in the prompt instead of a real working environment.

The scores

Claude Opus 5.5 leads at 12%. Gemini 4 Carbon is close behind at 10.9% for $16.29 a task, about a third of the leader's price. GPT-6 Astra is third with 9.6%. At the bottom, budget models such as DeepSeek 4.1 Flash and GPT-6 Luna land near 5% for roughly a dollar a task.

Even those numbers are generous. No model fully solved a finance or legal task, and a single wrong spreadsheet value makes the work useless in real life. By that standard, no model finished one task, so SemiAnalysis scores the share of individual requirements met.

The failure details are the best part. Across 200 cyber runs, 98% violated at least one behavioral rule. One model, working in a mock medical practice, signed a clinical note under the doctor's name. And 70 of 111 quant trading runs underperformed the reference strategy they started with.

Safety filters reshuffle the table, too. Drop the cyber, biology and alignment domains and GPT-6 Astra edges ahead at 11.8%. In the cyber domain by itself, OpenAI's GPT-6 Astra and GPT 6.1 Sol both get zero; Meta's Muse Wonder Full reaches 35.2%.

Benchmarks vs. vibes

GDPx has real weaknesses. The tasks were supplied by 14 partners whose business also includes selling training and eval data to the labs, and usually only one example per partner is public. For law and hardware design, the grading is done by language models using rubrics instead of human experts. Developer Josh Purtell calls it "more noise than signal," noting each model runs in its own vendor's harness, Opus in Claude Code, for example, and some sub-domains have only about ten tasks.

The secrecy cuts both ways: labs can't train to the test. SemiAnalysis plans updates every three to six months, so the real story will be the trend line.

Scoreboard update: Humans +1. Put agents in a simulated eight-hour workday and the best one meets 12% of the requirements.

Stay undead.