Google's StreamVibe tries to teach AI when to speak up, and shows how hard that is

Knowing when to say nothing is one of the hardest skills there is, for people and apparently for AI.

Researchers from Google, Stanford, Oxford and several US universities built StreamVibe, a video assistant meant to watch you work, say assembling furniture or following a software tutorial, and chime in only when you need it, as The Decoder's Jonathan Kemper reported. Today's assistants handle explicit commands like "tell me when the water boils" but miss unspoken goals, lose track of what's done and cling to instructions after users change them.

What they built

Their benchmark holds roughly 4,100 test cases drawn from 183-plus hours of daily-life and screen recordings; Gemini 2.5 Flash proposed when and what to say, and four human raters had to agree on every example. The model, StreamVibe-VL, apparently built on Qwen3.5-9B, makes a speak-or-stay-silent call once per second, with reinforcement learning to make it less chatty and a separate memory that tracks pending requests and finished steps.

Compared with other models, it was 10.7 percent better on timing and response quality, and the memory system added 12.3 percent on proactive help while cutting needless interruptions. Google's own Gemini 3.8 Flash still scored higher on several tests.

The reality check

The sobering result is about continuous use. When models had to process a whole video and build on their own earlier remarks, every system tested got much worse, with one key quality metric dropping by up to 61.5 percent. Past five minutes, they rarely picked the right moment and struggled to follow a changed request. The researchers concede real life is messier still. Proactive assistants are coming; good ones are not here yet.

Scoreboard update: Humans +1. AI still can't tell when to speak up over a long task.

Stay undead.