Building with AI models
Also called AI engineering, LLM application development, prompt engineering
The demo takes an afternoon. Everything after it is measurement — and the people who can measure are the ones who ship.
The claim This skill is not learned by learning the API; it is learned by building an evaluation set from your own failures, which is the one part nothing else will do for you.
Every other skill on this site was changed by these models. This one is made of them, which produces a peculiar learning problem: the thing you are studying answers your questions about itself, fluently, and is wrong in ways that read exactly like being right. The distinguishing skill of the field is not prompting. It is knowing whether your system got better.
The shape of the work is stable enough to teach even though the models are not. Something goes in, a model produces something, and you have to decide — before a user does — whether that something is acceptable. Almost every hard problem is downstream of that decision: retrieval exists because the model needs facts it does not have, evaluation exists because you cannot eyeball ten thousand outputs, guardrails exist because the input can be hostile, and cost and latency exist because the naive version of any of it is too slow and too expensive.
The practitioner literature converges on the same first move, and it is not the one beginners make. Look at your data. Read a hundred real outputs by hand, label the failures, and let the taxonomy of failures — not a framework's tutorial — decide what you build next. Hamel Husain's evals writing and the collectively written 'What We've Learned From A Year of Building with LLMs' both put manual error analysis before any tooling, because a metric chosen before you have seen your failures measures the wrong thing precisely.
The second thing to internalise early is that autonomy is a cost, not a feature. Anthropic's guidance on agents makes the point plainly: most production value comes from composable patterns — a chain, a router, a tool call — and agentic loops earn their unpredictability only on tasks where you genuinely cannot enumerate the steps. A beginner who reaches for an agent first spends their learning time debugging non-determinism instead of learning the pieces.
And there is a security dimension with no clean solution, which you must learn as a constraint rather than a chapter. Anything that reads untrusted text — a web page, a document, an email — can carry instructions to your model, and if that model can also act, you have handed a stranger your tools. Treat prompt injection as an architecture question about what the system is permitted to do, not as a filter you can add later.
What actually changed
-
WasMachine learning meant a training pipeline, a dataset and a research background.
NowThe model is an API call, so the scarce skill moved to the edges: what you feed it, what you let it touch, and how you know it worked.
-
WasSoftware was deterministic, so a passing test meant the behaviour was fixed.
NowThe same input can produce a different output tomorrow, so quality is a distribution you sample rather than a property you assert — which most engineers have never had to reason about.
-
WasA capable demo predicted a capable product.
NowA demo predicts almost nothing, because the demo is drawn from the cases you thought of, and the failures live in the ones you did not.
-
WasUntrusted input threatened your parser.
NowIt threatens your instructions. Prompt injection has no reliable filter, so the defence is what the system is allowed to do, not what it is allowed to read.
-
WasBenchmarks told you which model to use.
NowPublic benchmarks tell you almost nothing about your task; the only benchmark that predicts your product is the one you wrote from your own failures.
The path
In order. Each stage names what the model may and may not do while you are in it — that rule is the difference between the two arms of every study at the bottom of this page.
-
1 Call the model directly, with nothing in the way
Build an accurate mental model of the primitive before any framework hides it.
- Write a fifty-line script that calls a model API and prints the response. No framework, no wrapper.
- Learn the actual objects: messages and roles, system prompts, tokens, temperature, stop conditions, structured output, tool definitions.
- Measure one request: tokens in, tokens out, latency, cost. Then multiply by a thousand and look at the number.
- Break it on purpose — oversized input, malformed output, a timeout — and handle each.
What AI may do here Have the model explain its own API by all means, then check it against the documentation, which is the one source that is versioned. Do not start from a framework tutorial: the abstraction will be load-bearing in your head before you know what it abstracts.
You are done when You can state the cost and p95 latency of one call in your application, from memory, and say what happens when the response is not valid JSON.
-
2 Look at a hundred outputs
Replace opinions about quality with a labelled taxonomy of your own failures.
- Collect a hundred real inputs — real, not invented — and run them.
- Read every output and write one line about what is wrong with it. By hand. This is the stage everyone skips and the one that decides whether the rest works.
- Group the failure notes into categories, count them, and rank by frequency times damage.
- Turn the top three categories into concrete test cases with expected behaviour.
What AI may do here It may cluster your labels after you have written them. It may not write the labels: a model grading its own family's output shares its blind spots, and you are trying to find exactly the errors it cannot see.
You are done when You have a written failure taxonomy with counts, and you can name the most common failure without looking.
-
3 Build the eval, then change the system
Get a number that moves when quality moves, so improvement stops being a matter of impression.
- Turn the test cases into an automated set with a pass criterion per case — assertions where you can, an LLM judge only where you must, and validate the judge against your own labels.
- Record a baseline. Every change from here is measured against it.
- Add tracing so you can see the whole chain of a single bad request.
- Re-run the set on a new model version before you upgrade, not after.
What AI may do here It can write the harness and the assertions. It may not decide what counts as correct — if the definition of good comes from the model, your evaluation measures agreement with the thing you are evaluating.
You are done when You have made a change that improved your eval score, and one that you reverted because it did not.
-
4 Add retrieval, tools and the parts that fail
Connect the model to real data and real actions, and learn the specific ways each connection breaks.
- Build retrieval end to end and evaluate the retrieval separately from the generation — most bad answers are a retrieval failure wearing a fluent sentence.
- Give the model one tool, then two, and watch what it does with an ambiguous request.
- Handle the boring failures: rate limits, partial responses, retries that duplicate side effects.
- Read the OWASP list for LLM applications and go through your own system against it, especially anything that reads text you did not write.
What AI may do here Fine for the plumbing, which is well-trodden. Not for your trust boundary: draw by hand what the system may do without a human, because that diagram is your entire defence against prompt injection and a generated one will be reassuring rather than correct.
You are done when You can point at where untrusted text enters your system and say exactly which capabilities it can reach.
-
5 Run it in front of people
Learn the operational half — cost, latency, drift, and what users do that you did not imagine.
- Ship it to real users with logging you will actually read.
- Watch cost per user and latency per step; optimise the step that dominates, not the one that annoys you.
- Feed real failures back into the eval set weekly. The set is a living artefact or it is decoration.
- Re-run everything when the model changes underneath you, because it will.
What AI may do here Anything it proposes about your production behaviour is a hypothesis to check against your traces. The traces are ground truth; the explanation is a story.
You are done when Your eval set contains cases that came from real users and that you would never have thought of.
Keep these unaided
The gap between what you can do with the tool and what you can do without it is the only honest measure of whether you learned anything. Check it occasionally, on purpose.
- Write an evaluation case from a real failure, by hand, with a pass criterion someone else could apply.
- Read a trace and say which step went wrong.
- Compute the cost and latency of a request path on paper.
- Explain why a retrieval returned nothing useful, without guessing.
- Draw the trust boundary of a system that reads untrusted text and can take actions.
Where the model genuinely helps you learn
- Writing the harness, the parsing, the retries and the other plumbing you have already specified.
- Clustering failure labels you wrote yourself, and proposing edge cases you can then check for realism.
- Explaining an unfamiliar part of the stack — a vector index, a tokenizer, a streaming protocol — with the documentation open next to it.
- Playing adversary: ask it to attack your own prompt and see which attacks your architecture already prevents.
- Turning your own written notes into a first draft of documentation, which you then correct.
Traps
- Framework-first learning. You learn the abstraction, the abstraction changes, and you are left with no model of what was underneath.
- Vibes-based iteration — trying prompts until the demo looks good. It converges on the examples you happened to try and cannot survive a model upgrade.
- An LLM judge validated against nothing. It agrees with you, cheaply, forever, and its agreement is what you were trying to test.
- Agents as the default. Autonomy multiplies the number of places a failure can hide; the industry guidance is to reach for the simplest composable pattern that works.
- Trusting public benchmarks for your task. They measure something; almost never your thing.
- Treating prompt injection as a content filter problem. It is a permissions problem, and the filter will make you feel safer without being safer.
What counts as proof now
The artefact stopped being evidence — anyone can generate one. These are the things that still say something about you.
- An eval set with real cases, a baseline, and a history of changes that moved it.
- A written failure taxonomy from outputs you read yourself.
- A shipped system with cost and latency numbers you can defend.
- A post-mortem of one failure that reached a user, with what you changed.
Where to learn it
Courses
-
One to two hour hands-on courses, free, usually built with a vendor. Good for meeting a technique quickly; each is a tour, not a curriculum, and the vendor's tool is always the answer.
Books
- AI Engineering Paid
The systems-design view of building applications on foundation models — evaluation, adaptation, inference optimisation. The most complete book-length treatment; necessarily behind the frontier on specific models.
Free reading and references
-
The clearest statement of when a workflow beats an agent, with the patterns named. Written by a model vendor, so read the framing critically — but the advice is conservative in the direction that helps a learner.
-
Six practitioners writing down what actually worked, organised tactical to strategic. The best single document in the field, and honest about what remains unsolved.
-
The canonical argument for error analysis before tooling, with concrete workflow. Start here the moment your demo works and you cannot tell whether changes help.
- OpenAI Cookbook Free
Runnable recipes for the common patterns. Vendor-specific and it dates quickly; use it for mechanics, not for architecture.
-
The security checklist for this stack, prompt injection first. Terse; treat each item as a question to ask your own architecture rather than a control to buy.
-
Years of careful writing on prompt injection specifically, including why the obvious fixes do not work. The best antidote to security advice that sounds complete.
The evidence
What each source measured, and what kind of source it is. A practitioner study and a randomised trial are both worth reading and are not the same claim.
- Dell'Acqua et al. (2023/2025), Organization Science Peer-reviewed
758 consultants: large gains inside the model's capability frontier, and 19 percentage points worse than the unaided control just outside it. The reason your own eval set exists — capability is jagged and cannot be inferred from a demo.
- Lee et al. (2025), CHI Peer-reviewed
319 knowledge workers, 936 first-hand examples: higher confidence in the tool went with less critical thinking, higher confidence in oneself with more, and the work shifted from producing to verifying. Self-reported survey, not a performance measure.
-
Experienced developers were 19% slower with AI tools on their own repositories while believing they were 20% faster — the sharpest available demonstration that in this field self-assessment is not measurement.
The learning science underneath
None of the advice above is original to this skill. It is the general findings on how humans learn, applied to one job — and each of these pages sets out the evidence for the principle itself.