The Median Trap

What went wrong

You can read a dose-response curve straight out of execution history: group the jobs by how many skills they loaded, average the quality score in each bucket, plot it. A shape appears. It looks like a finding. It is not one, because nothing assigned those skills at random. A router, a heuristic, or a person decided — and those decisions track job difficulty. Hard jobs attract more scaffolding. Hard jobs also score worse. The curve is a picture of the selection process wearing the costume of an effect. When we randomised the assignment properly, the shape vanished.

Two details worth publishing. The first is that this page contradicted itself in plain sight: the essay claimed a dose-response curve, and the "Common questions" block at the bottom of the same page answered "Do skill checklists improve LLM agent output?" with the correct null — +0.002, inside the noise. Both were visible to any reader who scrolled. Someone corrected the answer and never corrected the essay above it, and nothing in our publishing process compared the two halves of one document. The second is that the retraction required no new evidence: the corrected claim was already recorded, already marked as not fit to publish, and already sitting in a database we own. What was missing was any step that checks the register before a page ships. A rule that exists and is never consulted is indistinguishable from a rule that does not exist.

A note on this correction itself, since the same discipline applies. The first version of this notice, published earlier today, said the contradiction was between the page's structured data and its prose — machine-readable metadata saying one thing, the visible article another. That was wrong, and it was wrong for the ordinary reason: it was written from a stale local copy of the site rather than the version actually live. Checked against what shipped, both figures were on the page and both were visible. The corrected account is the one above. We are leaving the trail rather than quietly editing it, because a correction that needs correcting is exactly the case the practice is for.

What survives is narrower and more interesting than the original claim. The null rules out the naive mechanism — more instructions, better output — which is what most agent frameworks are built on: a markdown file of rules, appended to a system prompt, unmeasured. Our reading is that the format is the problem rather than the quantity. A static instruction list cannot be retrieved on demand, cannot record whether it was ever used, cannot be ranked against its alternatives, and cannot be retired when it stops earning its place. It cannot be measured, so it cannot improve — and you end up arguing about how many to load because there is nothing else to argue about. The lazy-loading argument survives on its own terms: loading only what a task needs cuts tokens and cost, and that saving never depended on the quality question.

---

Below: the original article as published, April 2026. Retained for the record. Its figures are withdrawn.

When software agents fail, the instinctive reaction is to give them more instructions. We write a checklist of "skills" or "rules" — always handle null pointers, never use nested ternary operators, ensure database connections are closed — and append them to the system prompt.

We call this "skill loading." It feels like engineering. It looks like progress. But in our controlled testing, we discovered that loading more than three skills simultaneously into an agent's active context window actually degrades the quality of the generated code.

We call this phenomenon The Median Trap.

The Experiment

To measure the exact impact of skill loading on code generation, we designed a controlled experiment using Claude 3.5 Sonnet. We generated a set of 11 distinct coding tasks of moderate complexity (e.g., implementing a rate limiter, parsing a custom file format, writing a thread-safe cache).

We then ran these tasks across 88 total executions, varying only the number of "skills" (explicit code-quality rules) loaded into the system prompt:

• Control Group: 0 skills loaded (pure task description).
• Group A: 2 skills loaded.
• Group B: 5 skills loaded.
• Group C: 10 skills loaded.

Each output was evaluated by an automated test suite for functional correctness, and graded by an independent LLM-as-a-scorer for architectural elegance, adherence to constraints, and code cleanliness.

The Results

The data revealed a clear, non-linear relationship between prompt complexity and output quality:

• The Sweet Spot (1-3 Skills): Adding 2 highly relevant skills improved the architectural score by 14% compared to the control group. The model successfully incorporated the constraints without losing sight of the primary objective.
• The Cliff (5+ Skills): At 5 skills, performance began to revert to the baseline. At 10 skills, the architectural score dropped 22% below the control group.
• The Compliance Paradox: While Group C (10 skills) had the lowest overall code quality, it had the highest literal compliance with the checklist. The model spent so much attention budget avoiding the "forbidden" patterns that it wrote overly verbose, convoluted, and fragile code to satisfy the rules.

Why the Median Trap Happens

Large Language Models do not process instructions like a compiler processes code. They process them as attention weights.

When you load 10 different rules into a prompt, you are forcing the model to distribute its attention across 10 different dimensions of constraint. Because the model's capacity for reasoning per token is finite, it is forced to find the "median" path of least resistance.

Instead of writing the most elegant solution for the specific problem, it writes a generic, defensive solution that guarantees none of the 10 rules are violated. It optimizes for non-violation rather than excellence.

Escaping the Trap: Lazy Skill Loading

The solution is not to abandon code-quality rules. The solution is to change how they are loaded.

Instead of statically loading every skill your agent might ever need, you must load them dynamically and lazily. If a task does not involve database operations, the agent should not have database-connection rules in its context. If a task is a simple utility script, it should not be burdened with enterprise-grade logging constraints.

By keeping the active skill count below 3 at any given moment, you preserve the model's attention budget for the actual problem-solving task.

This is why we built and open-sourced context-steward. It acts as an automated gatekeeper, analyzing the current task context and injecting only the highly relevant skills on demand, keeping your agent's attention focused where it matters most.

FAQ

Common questions