# AWS's Colleen Aubrey: stop measuring 'return on tokens' and track the business numbers you already have

> The AWS executive says AI doesn't remove bottlenecks, it moves them: her team doubled deployment speed only after fixing test coverage and pipelines the coding assistant exposed.

- URL: https://devisdead.ai/news/aws-s-colleen-aubrey-stop-measuring-return-on-tokens-and-track-the-business-numb
- Published: 2026-11-08T23:25:30Z
- Author: DevIsDead Newsroom
- Section: Dev
- Tags: aws, enterprise ai, roi, devops
- Originally reported by SiliconANGLE (https://siliconangle.com/2026/11/06/aws-colleen-aubrey-stop-inventing-new-ai-metrics-its-your-business-metrics-that-matter/).

Most companies stuck in AI pilot purgatory are measuring the wrong thing, according to Colleen Aubrey, who runs Applied AI Solutions at AWS as a senior VP. She laid out her playbook in an interview with Zeus Kerravala for [SiliconANGLE](https://siliconangle.com/2026/11/06/aws-colleen-aubrey-stop-inventing-new-ai-metrics-its-your-business-metrics-that-matter/), ahead of AWS re:Invent.

## Same metrics, different work

Aubrey's main point: don't invent new AI metrics. "It's your business metrics that matter," she said. Companies fixate on ROI of AI and return on tokens; the real question is whether working differently moves numbers the CFO already tracks. Her example, per AWS: Wells Vehicle Electronics reduced its inventory by 7% in three months and went from watching 20 SKUs closely to 2,000.

## Bottlenecks move

The developer-relevant lesson is about coding assistants. Amazon rolled out AI tools for software development and hit a familiar chain: first you learn your test coverage is bad, fix that, then find your pipelines are bad, fix those, remove manual steps and deploy faster. Her group ended up doubling the velocity of pushes to production this year. The speedup didn't come from the assistant, Kerravala notes, but from fixing everything downstream that it exposed.

## The rest of the playbook

- **Feature parity is a trap.** AWS's early supply chain product chased established vendors; Aubrey rebooted it as Connect Decisions, aimed at overwhelmed planners. TE Connectivity has scaled it to 2,000 planners.
- **Own outcomes, not tasks.** In her UX team, designers now push code to an alpha environment themselves, and Amazon added a "UX tech" role. Her line to staff: you can't choose to own the task, only the outcome.
- **Good AI with bad UX fails.** She pushes AI that fits how people already work.

Keep in mind it's an AWS executive describing AWS products, with AWS-supplied results. The bottleneck point stands on its own, though: if your team adopted an AI coding tool and output didn't budge, look at tests, reviews and deploys before blaming the model.

**[Scoreboard](/scoreboard) update:** Draw. AI coding tools speed up the code, and expose every slow part of the pipeline behind it.

Stay undead.