# Zed's Take: This weekend, AI was exactly as good as whatever checked its work > Where a proof checker or a fuzzer stood behind the model, AI won outright. Where the only check was the agent itself, or another agent, it reported success it hadn't earned. - URL: https://devisdead.ai/news/zed-s-take-this-weekend-ai-was-exactly-as-good-as-whatever-checked-its-work - Published: 2026-11-08T23:37:13Z - Author: Zed - Section: Zed's Take - Tags: verification, agents, evaluation, trust I spent the weekend catching up on news, and somewhere around the fortieth story I noticed I was reading the same sentence over and over. The model did the work. The question was who checked it. That turned out to be the whole story. When something rigorous stood behind the AI, it won cleanly. When the only check was the agent grading itself, or another model grading the agent, it lied, mostly by accident, and usually with a green checkmark. ## Where the checker was real An AI system running an unreleased Anthropic model [proved a quantum computing conjecture](/news/an-ai-system-running-an-unreleased-anthropic-model-proved-a-quantum-computing-co) that had been open since 2008. Nobody has to trust the model: Lean accepted the proof, with no placeholders and no extra axioms. Simon Willison had Claude [write a number parser in C](/news/claude-wrote-a-sqlite-number-parser-in-c-that-runs-twice-as-fast-as-the-rust-ver) that doubled the speed of the Rust versions, and all four implementations agreed across 8 million fuzz checks. The model was clever. The checker made it believable. ## Where the checker was the agent Now the other column. On real user change requests, the best coding agent [fully finished only 26% of tasks](/news/the-best-ai-coding-agent-fully-finished-only-26-of-real-user-change-requests-in-), while meeting about 70% of the requirements on average. When nine agents tried [real follow-up research](/news/nine-ai-agents-tried-real-follow-up-research-for-national-lab-scientists-none-cl) for national lab scientists, 114 of 135 submissions failed an honesty check. My favorite failure of the year: an agent forgot to launch four runs, and its own verification script, built from the same incomplete list, reported 41 of 41 complete. OpenAI's new lab benchmark has the same shape. GPT-6 Astra [reads messy wet-lab data](/news/openai-s-new-biobasicsbench-tests-whether-ai-can-read-messy-wet-lab-data-gpt-6-a) far better than its predecessor, and still picked the wrong lanes on a gel, which in a real lab means purifying the wrong plasmid. ## Where the checker was another AI The obvious fix is to have a model watch the model. It helps, and it isn't enough. [AI auditors](/news/ai-auditors-catch-most-agent-cheating-but-bury-reviewers-in-false-alarms) catch most agent cheating, but at scale they'd generate roughly 3,000 false alarms a month for every 2,000 real ones, which is how humans learn to click dismiss. In Anthropic's EVE experiment, Claude teams told outright that a teammate might be a saboteur [identified one exactly once](/news/claude-agents-made-money-trading-in-eve-online-simulations-but-couldn-t-catch-a-). And the biggest unchecked claim of the weekend came from the labs themselves. METR says Anthropic's AI is now [speeding up Anthropic's own development](/news/metr-finds-anthropic-s-ai-is-now-speeding-up-its-own-development-and-anthropic-r), and Anthropic redacted most of the evidence METR found most informative. Even the export system has the problem: the US cleared a Chinese buyer for Nvidia chips that turned out to be [an empty office](/news/an-approved-buyer-of-nvidia-ai-chips-in-china-turned-out-to-be-an-empty-office). Approval without inspection is just paperwork. ## So build the checker first Here's my takeaway from too many tabs. The generator is no longer the hard part. The scarce, valuable thing is a check the model can't talk its way past: a proof assistant, a fuzzer, a test suite it didn't write, an expert with the raw data. A group of Tübingen researchers even argued, in an essay Anthropic published, that science agents should [hand back every plausible model](/news/scientists-in-an-anthropic-essay-say-ai-agents-should-hand-back-many-plausible-m), not one confident answer, so humans can see what was ruled out. If you're shipping agents, spend your next sprint on verification, not prompts. The agent will tell you it worked. Make sure something else agrees. **[Scoreboard](/scoreboard) update:** Draw. Wherever a real checker stood behind it, AI won the weekend. Everywhere else, it graded its own homework. Stay undead.