Is high AI token usage a good sign for a product team?

Not on its own. The same token count can come from a team doing excellent work or from one burning tokens with little to show for it. A token number only means something next to what it bought: more work shipped, a decision that changed, or a workflow that costs less than before.

How do you measure the value of AI tokens in delivery work?

Track the share of AI output a person still has to fix before it ships. In delivery the task is known, so the AI should be finishing more of the work over time. That fix rate should fall as the workflow matures.

How do you know if AI research is worth the tokens?

Before you start, write down the decision the research feeds, your current answer and the evidence that would change it. Afterwards, check whether you found that evidence or just more pages. Research that moves a real decision was worth its cost.

How can product teams cut AI token costs?

Measure each recurring workflow like ad spend, comparing a person's cost with the cost of the runs. Then use plain code where the rule is fixed, the smallest model that passes, and cached context with batched runs.

Who should own AI token spend in a product team?

Finance sees the bill and engineering sees the usage, but only the product manager can say what the spend was meant to buy. That makes judging its value a product job.

What should go next to a token number in a report?

A second column showing what the tokens bought. For delivery, that is more shipped and less fixed. For discovery, a decision that changed. For operations, a cost lower than the old way, counting people, tools and process.

Measuring AI Token Spend: 3 Questions That Show What Your Tokens Bought

1

The full infographic, as a PDF

The exact one from the post, sized to print or keep within reach.

2

A copy in your inbox

The download link lands in your email too, so it's there when you need it. No follow-up campaign.

3

The Product System, weekly

My note on leading product in the AI era. One idea a week that compounds. Unsubscribe anytime.

One OpenAI employee used 210 billion tokens in a week.

Hero or headache? The number won't tell you.

I see product teams reporting token spend like a win. But the same number could mean a brilliant team or a reckless one. On its own, nobody can tell which.

Three questions can. One for each kind of work tokens pay for: delivery, discovery and operations. Asking them is a product job, and this page walks through each one, then the simple reporting rule that ties them together.

Delivery: is the AI finishing the work?

In delivery work, the task is known. You know what done looks like: the ticket, the draft, the test, the release note. Tokens here pay for getting known work finished.

So the question is simple. Is the AI finishing the work?

The way to answer it is to track the share of the output a person still fixes. If every AI draft needs heavy rework before it ships, the tokens bought a first attempt, not a finished piece. That share should fall as the workflow matures. If it doesn't fall, the setup needs attention: the context, the instructions or the checks around the work.

A rising token count with a flat fix rate is not progress. A steady token count with a falling fix rate is.

Discovery: is this research worth the tokens?

In discovery work, the answer is unknown. That is the point of discovery. Here tokens can buy headspace to go deeper than a person could alone: more sources read, more angles tried, more options explored.

Or they can buy tokens going round in circles.

The question is: is this research worth the tokens? To answer it, write three things down before you start. The decision the research feeds. Your current answer to it. And the evidence that would change that answer.

Afterwards, check one thing. Did you find that evidence, or just more pages?

Research that changes or confirms a real decision was worth its cost. Research that produces a long document and leaves the decision where it was bought volume, not learning. Writing the decision down first is what lets you tell the difference.

Operations: how do you cut the cost?

In operations, the work is the same workflow, every day. A weekly report, a triage step, a data clean-up. Here the question is not whether to spend tokens, but how to spend fewer for the same result.

Measure it like ad spend: a person's cost against the run's tokens. If the workflow replaced a few hours of someone's week, compare that cost with what the runs cost. That gives you a number you can defend, and a baseline you can improve.

Then pull the levers.

Use plain code where the rule is fixed. If a step always follows the same logic, a model does not need to think about it every time.

Use the smallest model that passes. Not every step needs the most capable model. Test a smaller one against the same standard, and keep it if it passes.

Use cached context and batched runs. Repeated context and one-by-one calls add cost without adding value.

Why token spend is a product question

Different people see different parts of the picture.

Finance sees the bill. Engineering sees the usage. Only the PM can say what it was meant to buy.

That is why measuring AI token spend belongs with product. The bill and the usage logs show what went in. Only someone who knows the decision, the customer and the workflow can say what was supposed to come out, and whether it did.

The rule: every token number gets a second column

My rule is simple. No token number goes in a report on its own. It gets a second column: what it bought.

For delivery, the second column is more shipped, less fixed.

For discovery, it is a decision that changed.

For operations, it is cheaper than the old way, counting people, tools and process.

Tokens are the cost of the bet. The second column tells you if it paid off. That is the number worth celebrating.

This is the same problem as other numbers teams report with pride. The Museum of Meaningless Metrics at /vault/museum-of-meaningless-metrics looks at lines of code, story points, pull requests and tokens spent side by side, and the one question that sorts them.

Where do most of your tokens go: delivery, discovery or operations? If you want to see what AI can automate in your own PM work, you can check where you stand in 2 minutes at martongaspar.com/scorecard.

Questions people ask

Is high AI token usage a good sign for a product team?: Not on its own. The same token count can come from a team doing excellent work or from one burning tokens with little to show for it. A token number only means something next to what it bought: more work shipped, a decision that changed, or a workflow that costs less than before.
How do you measure the value of AI tokens in delivery work?: Track the share of AI output a person still has to fix before it ships. In delivery the task is known, so the AI should be finishing more of the work over time. That fix rate should fall as the workflow matures.
How do you know if AI research is worth the tokens?: Before you start, write down the decision the research feeds, your current answer and the evidence that would change it. Afterwards, check whether you found that evidence or just more pages. Research that moves a real decision was worth its cost.
How can product teams cut AI token costs?: Measure each recurring workflow like ad spend, comparing a person's cost with the cost of the runs. Then use plain code where the rule is fixed, the smallest model that passes, and cached context with batched runs.
Who should own AI token spend in a product team?: Finance sees the bill and engineering sees the usage, but only the product manager can say what the spend was meant to buy. That makes judging its value a product job.
What should go next to a token number in a report?: A second column showing what the tokens bought. For delivery, that is more shipped and less fixed. For discovery, a decision that changed. For operations, a cost lower than the old way, counting people, tools and process.