We randomised our own prompt scaffolding across 88 runs. It did nothing.

Skill checklists are prompt scaffolding: reusable instructions injected into an agent's context to tell it how to do a job well — naming conventions, security rules, code style. The premise is that better instructions produce better output.

We ran the experiment. Across 88 scored executions with the checklists randomly assigned, output quality moved by +0.002. That is inside the noise. No measurable lift.

We are publishing this because we built the checklists, believed in them, and measured them anyway.

What we actually tested

Our platform runs a native skills system: a job can pull scaffolding — security rules, naming conventions, code style, documentation standards — into the model's context before it writes anything. Each execution records which skills it loaded.

The obvious way to evaluate that is to look at your history: compare jobs that loaded skills against jobs that didn't. Do it and you will find an effect. The effect will be fake.

It is fake because nothing assigned those skills at random. A human, a router, or a heuristic decided — and those decisions track job difficulty. Hard jobs attract more scaffolding. Hard jobs also score worse. The scaffolding looks harmful, or helpful, depending on which way the selection ran. You are measuring who got the treatment, not what the treatment did.

So we randomised. Each job in the sweep was assigned its arm by a deterministic hash of the job ID — reproducible, balanced, and impossible to cherry-pick after the fact. Anyone with the job IDs can recompute the assignment and check we didn't move the goalposts. Then we scored the output with EQS, our open-source grader.

Result: 0.002. Eighty-eight runs. Nothing.

Why we shipped the null instead of burying it

A negative result is worth more than the feature it kills.

The checklists cost tokens on every single execution. They cost engineering time to write and maintain. They carried an assumption — good instructions produce good code — that felt so obviously true nobody had asked it to prove itself. It didn't.

What survives is not the checklists. It is the measurement: arm-randomised intervention testing, wired into a production platform, so the next idea that feels obviously true has to earn its place. That machinery is the actual asset. The null is just the first thing it caught.

We would rather run this than tell you our agents are smart.

The boundary — read this before you quote us

This result is about our prompt scaffolding, inside our platform, scored by our grader.

It is not a finding about Anthropic's Claude Skills, or any other vendor's skills feature. Same word, different thing. We have no data on those and we are not going to imply that we do.

It is also not a claim that context never helps. It is a claim that this intervention, measured this way, did not move the number.

What we still can't tell you

• How large an effect we could have detected. 88 runs is a small sample. A tiny real effect could hide inside it. What we can say is that nothing large enough to justify the token cost showed up.
• Whether it holds for other task types. The sweep ran on our workload. Yours is different.
• Whether more context helps. We tried to answer this from our history and caught ourselves making exactly the mistake described above — the apparent effect was an artefact of who got the treatment. That question is now blocked behind a randomised experiment we're running properly. When it finishes, we'll publish the answer whichever way it lands.

Questions

Every number here comes from our execution data and is registered in our claim registry. Where we don't have a receipt, we say so. Related: The Median Trap · We Deleted Our Own Quality Checks