Product design
Also called product management, product thinking, discovery, product strategy
Deciding what to build and finding out whether it worked. A model writes the plan, the persona and the case for it, none of which touched reality.
Listen to product design 33 episodes, about 29 h 30 minThe claim Every artefact product work runs on is now free to produce and reads as finished, so the only part left that is scarce is contact with reality: which problem, for whom, and did the number move.
Product design is the sibling discipline to interface design and it answers a different question. Interface design asks whether the screen is right. Product design asks whether the screen should exist — which problem it addresses, whose problem that is, what you are giving up to build it, and how you will know afterwards whether anything changed. The two get confused because they are often the same job title, and because both used to be bought with the same currency: years of watching decisions play out.
The discipline's central fact is unflattering and well measured. Most product ideas do not work, including the ones experienced people are confident about. Microsoft's own experimentation platform found that only about one third of tested ideas improved the metric they were designed to improve, and the write-up gathers the same finding from everywhere else anyone has looked — second-hand in each case, and worth reading as a pattern rather than as five independent measurements: Netflix reported as treating 90% of what it tries as wrong, Google as having about a tenth of its experiments lead to a business change, Etsy's own engineers writing simply that nearly everything fails. This is not a story about incompetence. It is what it feels like to work on a system too complicated to predict, and it is the reason the job is organised around finding out rather than around deciding well.
What got cheap is every artefact that represents the finding-out. A model will produce a product requirements document, a competitive teardown, a persona set, a roadmap and a confident argument for any of them, in a minute, in the register of someone who has done the work. The output of product thinking is now free; the input never was. A generated persona is a claim about people nobody spoke to, and its characteristic failure is not being crude — it is being agreeable. When Nielsen Norman Group compared simulated research participants against real ones, the simulated ones were systematically optimistic, listed every need without ranking any, and produced no behaviour at all, because they had never used anything.
Meanwhile the thing these documents are supposed to prevent keeps happening. Pendo measured feature usage across 615 of its customers' products and found that 80% of features are rarely or never used, with about 12% of features generating 80% of daily usage. Everything in that 80% was specified, argued for, prioritised, designed and built by people doing their jobs. Shipping is not evidence. It was never evidence, and now that the case for shipping is free to generate, it is less evidence than before.
So the path below is organised around the few things that still cost something: stating a problem in a form that could turn out to be wrong, getting your own evidence from people who do not want to help you, deciding in public what you will not do, and reading a result that says you were mistaken. A model is useful at every one of those stages and it can replace none of them, for a reason worth being precise about — it can generate any position you ask it for, which makes it an excellent sparring partner and a worthless witness.
What actually changed
-
WasA requirements document took a week, and the week was the point: you could not fill the pages without finding things out.
NowIt takes a minute and arrives complete, so the finding-out has no natural home in the process and quietly stops happening.
-
WasA competitive teardown meant a day of actually using four competitors, which is where the surprises lived.
NowA summary assembled from what has been written about those products arrives instantly — accurate about their marketing, silent about their onboarding.
-
WasArguing for a feature cost you political capital, so you only spent it on arguments you believed.
NowThe argument is free to produce and reads as reasoned, so the cost that used to filter out weak ideas is gone and the filter has to be rebuilt deliberately.
-
WasPersonas were a compression of research somebody did, and were only as good as that research.
NowThey are generated on request, sound like people, and fail in one direction: too agreeable, too complete, and incapable of surprising you.
-
WasDeciding not to build something was defensible because the alternative was visibly expensive.
NowBuilding it looks cheap, so the discipline of saying no has lost its best argument and needs a better one: attention and support cost the same as they always did.
The path
In order. Each stage names what the model may and may not do while you are in it — that rule is the difference between the two arms of every study at the bottom of this page.
-
1 State problems that can be wrong
Learn to write the thing the whole job hangs on: a problem statement that somebody could disagree with.
- Take the last three things you or your team built and write, for each, the one-sentence problem it was meant to solve. Notice how many are descriptions of the solution.
- Rewrite each until a reasonable colleague could say 'I don't think that's true' — which means it names who, in what situation, and what they currently do instead.
- Ban the words seamless, intuitive, engagement and streamline from your drafts and see what is left.
- For each statement, write the observation that would prove it false.
What AI may do here Use it to attack the statement, never to write it. Ask for the three strongest reasons your problem statement might be false, then go and check them yourself. A statement the model wrote will be fluent, general and unfalsifiable, because those are the properties of text that sounds right to everyone.
You are done when You can hand a problem statement to someone outside your team and they can tell you whether they believe it, without asking what the product is.
-
2 Get evidence nobody generated for you
Buy the one input that has no substitute: real people behaving in ways you did not predict.
- Talk to five people who have the problem, weekly, and keep doing it. The interval matters more than the number.
- Ask what they did last time, never what they would do. Past behaviour is data; predicted behaviour is fiction told politely.
- Write your prediction before each conversation and score it afterwards. The gap is your finding.
- Watch someone use the thing in silence for ten minutes. Say nothing, including when they go wrong.
What AI may do here No synthetic participants, at any stage, for any reason — not as a pilot, not as a warm-up. Use it to draft your question list and to cluster your own transcripts, then reread the transcripts yourself, because the summariser drops the outlier and in discovery the outlier is frequently the entire finding.
You are done when You have three written observations from real people that contradict something you believed a month ago. If everything you heard confirmed your plan, you were presenting, not interviewing.
-
3 Decide in public, and write down what you dropped
Turn a decision into an artefact that can be reviewed later, including the option you rejected.
- Write a one-page decision memo: the problem, the options you considered, the one you chose, what would make you reverse it.
- Do the arithmetic before the design. Reach, times the rate it applies, times the effect you expect — on paper, in public. Most ideas die here, cheaply.
- Say no to something out loud, to the person who wanted it, with your reason.
- Keep the memos. In six months they are the only record of what you actually believed.
What AI may do here It may pressure-test the memo and it may not write it. Ask it to argue for the option you rejected as hard as it can; if that argument is better than yours, you have learned something before shipping. What it must never do is supply the numbers in your arithmetic — a plausible estimate inserted where you did not have one is indistinguishable from a real one a month later.
You are done when Someone can read one of your memos, disagree with the decision, and point at exactly which sentence they disagree with.
-
4 Ship small, predict the number first
Close the loop that turns opinion into knowledge, in the only order that works.
- Write the metric you expect to move, the direction, and roughly how much — before release, where others can see it.
- Ship the smallest version that could move it. A version too small to measure is too small to learn from; a version too large to attribute is worse.
- Instrument the flow before you ship, not after you wonder.
- Keep the prediction next to the result, permanently.
What AI may do here Let it write the instrumentation, the copy variants and the release notes. Do not let it pick the metric: a model asked for a success metric will offer one that is easy to move, because that is what most writing about metrics contains. Picking a measure that can actually go down is the judgement being trained here.
You are done when You have one shipped change where you predicted the direction correctly and one where you did not, and you can say what you believe now that you did not believe before.
-
5 Read the results you did not want
Build the habit that separates people who learn from this job from people who accumulate years in it.
- Read every result, including the boring ones, and especially the flat ones. Flat is the most common outcome and the least discussed.
- Write one sentence per result about what you now believe. Not what you will do — what you believe.
- Kill something you championed, in public, citing the number.
- Once a quarter, reread your own memos from six months ago and mark the ones that were wrong.
What AI may do here Never let it interpret a result for you, and be specific about why: asked to explain a number, it will produce a coherent story for whatever the number is, including a story for noise. Reading a result you predicted wrongly is the highest-value minute in the whole discipline, and it is the one minute that delegating destroys entirely.
You are done when You can name three beliefs you have changed because of a measurement, with the measurement, and at least one of them cost you something to admit.
Keep these unaided
The gap between what you can do with the tool and what you can do without it is the only honest measure of whether you learned anything. Check it occasionally, on purpose.
- State a problem in one sentence that a reasonable person could dispute, without using a single fashionable adjective.
- Run a conversation with a user and come away with something that contradicts you.
- Do the reach-rate-effect arithmetic on the back of an envelope and conclude that an idea cannot matter enough to build.
- Decide not to build something, and defend that to the person who asked for it, using a reason rather than a process.
- Read an experiment result that says your idea lost, and say out loud what you now believe.
Where the model genuinely helps you learn
- As the opposition: have it make the strongest possible case against a decision you have already made, then check that case against your own transcripts and numbers. Defending your position out loud is retrieval practice, not delegation.
- As a transcript assistant on research you personally conducted — clustering, tagging, finding the quote you half-remember — provided you reread the source before you believe any cluster.
- As a fast way to learn the shape of a domain you are new to, treated as a list of things to verify rather than as facts: it is reliable about what is commonly written and unreliable about what is true.
- As a producer of the artefacts that follow a decision rather than precede it — release notes, tracking plans, spec text, the fourth variant of a microcopy string.
- As a rehearsal stakeholder: let it play the executive who wants the feature you think is wrong, and practise the conversation before you have it.
Traps
- Mistaking output for outcome. Pendo's measurement across 615 products found 80% of features rarely or never used — all of them specified, argued for and shipped by people who believed in them. A list of what you delivered is not a record of what you achieved, and it is the easier list to write.
- The plausible document. A generated requirements doc or strategy deck has the texture of a decided thing, and it ends arguments socially rather than by settling them. Fidelity arriving before evidence is the same failure as a polished mockup arriving before direction, moved one stage upstream.
- Synthetic users. They answer immediately, sound reasonable, and skew optimistic: NN/g found simulated participants claiming they finished courses that real participants admitted abandoning. They cannot be surprising, and being surprised is the entire product of research.
- Asking the model to validate. It will argue for whatever position you hand it, competently, which means agreement from it carries no information at all. Every validation you receive this way feels like evidence and is literally free to obtain, including for ideas that are wrong.
- Judging yourself by velocity. Shipping a lot feels exactly like being good at this, and the base rate says about a third of it does anything. If everything you ship appears to work, the thing that is broken is your measurement, not your intuition — and that conclusion is very hard to reach from the inside.
What counts as proof now
The artefact stopped being evidence — anyone can generate one. These are the things that still say something about you.
- Three decision memos written before the outcome was known, each naming the option you rejected and what would have changed your mind.
- A shipped change with the prediction you recorded beforehand and the number that followed, including one where they disagree.
- Something you killed, with the evidence that killed it and what it cost you to say so.
- An interview transcript next to the specific thing you changed because of it.
Where to learn it
Books
- Shape Up Free
One company's complete method for choosing and bounding work, readable free in the browser. Strongest on appetite and on shaping before committing; it is a single opinionated system from a company with unusual constraints, so read it as a worked example rather than a standard.
-
The most usable single book on keeping contact with customers as a weekly habit rather than a project. Light on strategy and on what to do when nobody will talk to you.
- Inspired Paid
The default text on how strong product organisations are structured and what the role actually is. Better on teams and principles than on technique, and its examples skew to large well-funded companies.
-
Why organisations measure themselves by output and how that is unwound. Aimed at people with some organisational leverage; if you are one person on a team it will describe your problem accurately and hand you few levers.
-
The reference for running experiments that mean something, from people who ran them at Microsoft, Google and Airbnb. Read it when you already have traffic and a result you do not trust.
Free reading and references
-
Two decades of free essays on product teams, discovery and the difference between feature teams and empowered ones. Assertive and lightly evidenced — useful vocabulary and organisational argument, not measurement.
- Product Talk Free
The most concrete free writing on continuous discovery: interview structure, opportunity mapping, weekly rhythm. Best read while you have interviews scheduled; it is a practice, and reading it without practising is the trap this section keeps warning about.
-
The argument that dissolves the most common excuse for doing no research. Note the argument is about usability problems in one design, not about market questions or preference — it is routinely over-applied.
Papers
-
Twelve free pages that contain the discipline's most important number and the pitfalls that invalidate most experiments. Written for an encyclopedia, so it is dense and assumes you want the statistics.
-
Usage measured across 615 products rather than surveyed. A vendor's report about a problem its product sells against, which is a reason to read the method section, and the method is straightforward.
The evidence
What each source measured, and what kind of source it is. A practitioner study and a randomised trial are both worth reading and are not the same claim.
-
Only one third of the ideas tested on Microsoft's experimentation platform improved the metrics they were designed to improve, and success is harder still in well-optimised domains like Bing. The same section collects comparable figures reported elsewhere, including Netflix treating 90% of what it tries as wrong.
- Pendo (2019), Feature Adoption Report Industry report
Product usage analysed across 615 Pendo subscriptions: 80% of features in the average software product are rarely or never used, and an average of 12% of features generate 80% of daily usage volume.
- Rosala and Moran, NN/g (2024) Practitioner study
AI-generated research participants compared against real ones: the synthetic participants were systematically optimistic, claiming course completions that real participants admitted abandoning, listed needs without prioritising them, and produced no behavioural data.
- Dell'Acqua et al. (2023/2025), Organization Science Peer-reviewed
758 BCG consultants: inside the model's capability, GPT-4 users completed 12.2% more tasks 25.1% faster at 40.2% higher rated quality — and on a task just outside that capability they were 19 percentage points more likely to reach the wrong answer than consultants with no model at all.
- Bastani et al. (2025), PNAS Peer-reviewed
Nearly 1,000 high-school students: unrestricted GPT-4 raised performance 48% while available and left them 17% worse than the control group once it was removed. A version constrained to teacher-designed hints erased the harm.
The learning science underneath
None of the advice above is original to this skill. It is the general findings on how humans learn, applied to one job — and each of these pages sets out the evidence for the principle itself.