How should a product manager respond to a "just add AI" request?

Don't say no, and don't say yes yet. Turn the vague ask into the right build by working through the questions behind it: the real customer problem, whether AI beats the simplest fix, what the AI owns, the context it needs, the quality bar, how failures are handled, who owns it and the hidden costs.

What should you check before shipping an AI feature?

Check the most costly failure modes and how to contain them, a fixed evaluation set that every version must pass, a human fallback for when the AI gets it wrong, a named owner after launch, and the hidden costs of speed, upkeep, security and trust. The pass mark should already have been set before the build started.

What is a non-AI baseline?

It is the current way of doing the job, or the simplest fix that doesn't use AI, such as a rule, a workflow change or a plain dropdown. Comparing the AI feature against it shows whether AI improves the outcome enough to justify its extra cost and risk.

What is an evaluation set for an AI product?

An evaluation set is a fixed collection of real and edge cases that every version of an AI feature must pass. Because it stays the same while prompts and models change, it turns "this version feels better" into "this version is better".

Why does an AI feature need a human fallback?

Because the AI will get some answers wrong. A human fallback defines when the AI stops, who takes over and how the user carries on, so a failure doesn't leave the user stuck.

How does skipping these steps affect a product manager's growth?

The steps form the feedback loop that turns junior execution into senior judgement. Each one forces a decision that can later be checked against what happened. Skip them and a team can ship quickly while the judgement that would make them senior never develops.

Just Add AI: 11 Questions to Answer Before You Build or Ship an AI Feature

1

The full infographic, as a PDF

The exact one from the post, sized to print or keep within reach.

2

A copy in your inbox

The download link lands in your email too, so it's there when you need it. No follow-up campaign.

3

The Product System, weekly

My note on leading product in the AI era. One idea a week that compounds. Unsubscribe anytime.

"Just add AI" sounds simple. It isn't.

If you have ever been handed an AI idea, this is for you. Your job isn't to say no. It is to turn a vague ask into the right build.

The person asking means well. They just can't see the work hiding behind "just". To them it is one small thing. In reality it is a stack of decisions and costs nobody has looked at yet.

That gap is where product managers earn their keep. The request often arrives from someone senior, with energy behind it and a deadline attached. Saying no outright closes the conversation. Saying yes outright commits the team to a build nobody has scoped. The useful move sits between the two: take the ask seriously and make the hidden work visible, one question at a time.

There is a second trap, and it shows up after the yes. A slick AI demo, an excited room, and a ship by Friday. That is exactly when a team skips the steps that build its judgement. The product managers shipping AI fastest can fall the furthest behind. Not in output. In judgement.

So here are 11 questions in two groups: five to answer before you build, and six to answer before you ship.

Before you build: is this worth building with AI?

1. What is the real customer problem?

"Add AI" is a solution looking for somewhere to go. Before anything else, write down whose problem this solves, what it costs them today, and what proves it matters. If nobody can say, the request isn't ready to build.

A good demo is very persuasive evidence that the technology works. It is no evidence at all that anyone needs it. Every later question depends on this one, because you can't judge the right tool, the right data or the right quality bar for a problem nobody has named.

2. Does AI beat the simplest fix?

Rule out a rule, a workflow change and a simpler fix. Plenty of problems that arrive labelled as AI can be solved with logic you write once, or a change to how the work flows. Often a plain dropdown wins.

Compare the AI idea against two things: the way the job is done today, and the simplest fix that doesn't use AI. If the simple version solves the problem, the AI version has to earn its extra cost and risk.

3. What does the AI own?

Pick the calls it makes, and the ones it never touches. Draw the line before you build, not after something goes wrong. Which decisions is the system allowed to make on its own, and which always stay with a person? Writing this down early also gives everyone the same picture of what is being built, which saves an argument later.

4. What context does it need?

A model can only work with what it is given. Feed it the data and rules it needs to be right. If the information lives in people's heads, in scattered documents or in a system nobody has access to, that gap is part of the build.

5. What must improve, and by how much?

Set the pass mark before you build, or every version will feel better and none of them will be. AI output varies from run to run, which makes "it looks good" an easy bar to clear and a poor one to ship on.

Agree the bar with the person who asked for the feature. It turns a vague request into a target you both recognise.

Before you ship: what happens when it gets it wrong?

6. What are the most costly failures?

Every AI feature will get some answers wrong, make some up and refuse others. Name the three failures that would cost you most, and how you will prevent or contain them. Your users find the rest, in public. Naming the worst cases up front is what lets you design for them instead of apologising for them.

7. Which cases must every version pass?

Build an evaluation set: a fixed collection of real and edge cases that every version has to pass. The set stays the same while the prompts and models change, so you can see whether a change helped or just felt like it did. That is how "feels better" becomes "is better".

8. What happens when the AI stops?

Decide when the AI hands over, who takes over, and how the user carries on. It will get things wrong. The question is whether the user is left stranded when it does.

9. Who owns it after launch?

An AI feature is not finished at launch. Someone has to keep the prompts, the data and the quality working as the product and the model change underneath it. If that person isn't named before launch, the job falls to whoever notices first that the answers have got worse, which is often a customer.

10. What are the hidden costs?

Budget for speed, upkeep, security and trust. Each one has a price that doesn't show up in the first demo. A feature that is slow, expensive to run, risky with data or quick to lose users' trust can cost more than it earns.

11. Is it only worth doing because it is AI?

This is the question that tests all the others. Take the AI out of the pitch. If the idea still solves a real problem, keep going. If the only thing left is the word, drop it.

Why these steps build senior judgement

The questions protect the launch. They also do something less visible.

Each one forces a decision that someone has to own and later check against what happened. Defining the problem, setting the bar, testing against a baseline, naming failures, building an evaluation set and planning the fallback: doing that over and over is how a product manager learns what good looks like. It is the feedback loop that turns junior execution into senior judgement.

Let a team skip the steps, and that loop never closes. The team ships something that looks finished, while the judgement that would make them senior never gets built. Their output climbs. Their judgement stays where it was. For more on that gap, /vault/generating-is-not-building covers why generating something is not the same as building it.

Use the questions as a sign-off ritual

Next time someone hands you a "just add AI" request, or your team brings you an AI feature to sign off, run through the questions out loud and let the team answer. The aim isn't to slow every launch down. It is to make sure the steps that build judgement happen on every launch, even the ones that come with a great demo.

It is worth asking which of the 11 your team skips most often. That one is your starting point.

AI is rarely the hard part. Everything around it is.

If the request comes from the top, When Your CEO Has a Bad Idea (/vault/when-your-ceo-has-a-bad-idea) has seven questions to ask before you say no. If you're curious how AI-native your own product judgement is, the free scorecard at martongaspar.com/scorecard shows where you stand.

The headaches image that went with this piece was inspired by Kasra Jadid Haghighi.

Questions people ask

How should a product manager respond to a "just add AI" request?: Don't say no, and don't say yes yet. Turn the vague ask into the right build by working through the questions behind it: the real customer problem, whether AI beats the simplest fix, what the AI owns, the context it needs, the quality bar, how failures are handled, who owns it and the hidden costs.
What should you check before shipping an AI feature?: Check the most costly failure modes and how to contain them, a fixed evaluation set that every version must pass, a human fallback for when the AI gets it wrong, a named owner after launch, and the hidden costs of speed, upkeep, security and trust. The pass mark should already have been set before the build started.
What is a non-AI baseline?: It is the current way of doing the job, or the simplest fix that doesn't use AI, such as a rule, a workflow change or a plain dropdown. Comparing the AI feature against it shows whether AI improves the outcome enough to justify its extra cost and risk.
What is an evaluation set for an AI product?: An evaluation set is a fixed collection of real and edge cases that every version of an AI feature must pass. Because it stays the same while prompts and models change, it turns "this version feels better" into "this version is better".
Why does an AI feature need a human fallback?: Because the AI will get some answers wrong. A human fallback defines when the AI stops, who takes over and how the user carries on, so a failure doesn't leave the user stuck.
How does skipping these steps affect a product manager's growth?: The steps form the feedback loop that turns junior execution into senior judgement. Each one forces a decision that can later be checked against what happened. Skip them and a team can ship quickly while the judgement that would make them senior never develops.