# Nearly half of what researchers 'know' about LLMs has already expired, a Stanford study finds

> Of 50 widely cited claims tested on 20 models, 44% no longer hold. Claims about what models can’t do die fastest, with an average lifespan of about 16 months.

- URL: https://devisdead.ai/news/nearly-half-of-what-researchers-know-about-llms-has-already-expired-a-stanford-s
- Published: 2026-11-04T11:19:21Z
- Author: DevIsDead Newsroom
- Section: AI
- Tags: research, stanford, evaluation, hallucinations
- Originally reported by The Decoder (https://the-decoder.com/rapid-llm-development-is-outrunning-the-research-meant-to-explain-it-study-finds/).

"LLMs can't count the r's in strawberry." "They lose information in the middle of a long context." If you've repeated either line in a meeting this year, you may be quoting an expired finding.

Stanford researchers Owen Queen and James Zou took 50 popular, often-cited LLM findings and retested them on 20 models from eight developers, released between July 2024 and September 2026. [The Decoder](https://the-decoder.com/rapid-llm-development-is-outrunning-the-research-meant-to-explain-it-study-finds/) reports that 44% of the claims have expired, meaning the original effect has largely gone. Another 34% have weakened a lot. Only 16% still hold, and 6% have grown stronger.

## Weaknesses go first

For the 22 findings they could date, the average lifespan was 15.7 months. After one year, 16% had expired. After two, 39%.

Claims about what models can't do disappear fastest. Five of the nine reasoning findings have expired, and none held steady or got stronger. That set includes the strawberry letter count, arithmetic in unfamiliar number systems, the lost-in-the-middle problem and caving on correct answers when users push back. The authors think these die quickly because each has a right answer, which gives labs an obvious target to train toward.

Tendencies last longer. Among the 11 findings that held or strengthened are models favoring their own answers when acting as judges, sycophancy toward users, a political lean to the left, open-ended questions getting almost the same answer every time, and lopsided "random" numbers.

## Hallucinated citations: mostly fixed, not everywhere

In a 2023 study, GPT-3.5 made up 55% of its scientific references. Sent the same prompts, GPT-6.1 Sol fabricated none and Claude Sonnet 5.5 about 1%. Yet Qwen3.7-Flash, released this year as well, still made up 69%. Claude Sonnet 5.5 also still fails around 10% of two-hop questions where it knows both facts, down from the 43% first reported for ChatGPT.

Models from the same few months vary widely, too. DeepSeek V4 Pro shows 11 expired findings while GPT-6.1 Sol shows 30, and release date, provider and size don't predict it.

## What to do with this

The authors' recommendation is good engineering hygiene: treat any finding about LLMs as a statement about one model at one point in time, and record the model, version and date, with prompts and code, so it can be rerun.

For developers, the practical version is to stop designing around folklore. That guardrail you built because "models can't do X" may be guarding against a problem your current model doesn't have, while the persistent ones, like a model grading its own work too kindly, may still be there. Retest your assumptions when you switch models. Your evals are the only claims that count.

**[Scoreboard](/scoreboard) update:** AI +1. Most of the well-known AI weaknesses researchers cite have already been trained away.

Stay undead.