Learning from the files people wrote for their agents

To hand part of your job to a model you have to write down what you would otherwise just do. The model fails visibly when a step is missing, so you keep adding steps until it stops failing. Thousands of people are now doing this, in public, and what they are producing is not a blog post or a course. It is a working professional's standard — in the imperative, with the numbers still in it, free, because it was written to be executed rather than sold.

That is a strange and good thing to be able to read, and it is also a very easy thing to read badly. These pages are about doing it properly: what the research says about why such a file is unusual, why one of them is never enough, and the order you have to work in if reading rules is going to leave anything behind.

Why a file written for a machine is worth your evening

There is a field that studies how to get expert knowledge out of experts and into training. Its founding observation is uncomfortable: people who are good at something cannot tell you how they do it. The standard estimate is that experts are unaware of about 70% of their own decisions, because the decisions became automatic on the way to becoming good. Ask directly and you get a confident, fluent, incomplete answer.

The measurement behind that is worth knowing precisely. Six expert programmers were recorded solving hard troubleshooting problems and then asked to explain what they had done, using several different interview methods. No single one of them could report more than 41% of their own diagnostic actions. But when the researchers pooled all six accounts, coverage rose to 87%. Nobody had the method. Between them, they nearly did.

Instruction built from this kind of painstaking elicitation measurably beats instruction built the ordinary way — an overall effect of g = 0.871 across domains, from a small enough set of studies that the authors say so themselves, and in surgical training a standardised mean difference of 1.36 on procedural knowledge. A skill file is an informal, unreviewed, self-administered version of that interview, conducted by a machine that keeps failing until the author fills in what they left out. It is not as good. It is the same kind of artefact, and there are thousands of them.

And why reading them will probably teach you nothing

These files read beautifully. A good one is dense, specific and confident, and getting through it produces a strong sense of having improved — which is exactly the sensation that predicts nothing about what you will still know next week. Worse, a rule is a compressed failure and the compression throws the failure away. Someone who reads always 0.96 has acquired a number. The person who wrote it has watched a hundred buttons at 0.9 look broken, and only one of the two can handle a case the rule does not cover.

So every curriculum here is built in the one order that survives contact with the research: you make the thing first, unaided and time-boxed, and only then open the file and grade yourself against it. What each page adds is the column the file does not have — the reason behind the rule, and an honest note where the reason turns out to be that one practitioner decided and no study exists.

Design

The field with the densest public corpus, because visual judgement is the hardest thing to hand to a machine and so the people handing it over had to write the most down.