Architecture · August 2026

You Cannot Out-Coach the Scaffold

Benjamin Taini · Founder, Bouletteproof

We put a mechanical gate in front of our plan generator and watched violations fall from nine to zero across seven generations. The gate never loosened once. The interesting part is where the last two defects lived.

A grounding gate is a mechanical check that verifies every file and symbol a generated plan references against the actual codebase before the plan is accepted. Our agent orchestration platform generates work plans — which files to create, which to edit, which symbols the changes will touch. Before the gate existed, the generator produced a plan that opened by creating a file that already existed in production. Nothing in the plan was checked against anything real. That plan was persisted, and only a human review stopped it.

So we built the gate: four rules, each one a check that can mechanically say no. A file created must not already exist. A file edited must exist. A symbol referenced must be defined somewhere, or declared by a sibling task that creates it. A single edit must stay inside a single region. Then we ran the same planning goal through the gated generator, over and over, and treated every rejection as data. This piece extends the 93/7 result — the finding that platform structure, not model choice, explains most of the quality variance — and it rhymes with our null result on checklist coaching. The numbers below are the full run, including the round that went backwards.

Seven generations, one constant

The gate was never loosened. Every improvement came from making the candidates smarter. Each generation below is the same goal, freshly planned, mechanically checked. The violation count is what the gate rejected; the change column is the one thing we altered before the next attempt.

Generation Violations What changed before the next run
1 9 Baseline: the gate turned on for the first time over an uncoached generator.
2 7 The authoring skill page grew from 354 bytes to about three kilobytes of explicit rules.
3 4 Gate fix: symbol lookups widened to the whole codebase — cross-file references had been over-rejected.
4 1 Refine loop: on rejection, the violation report is fed back and the generator gets one corrected attempt.
5 2 Went backwards: fresh candidate sampling varies. Published here because the wobble is part of the data.
6 1 A naming-conventions page taught the generator not to invent variants of existing symbol names.
7 0 Two one-line fixes — one in the gate's own input handling, one in the generator's scaffold. See below.
the gate — unchanged for all seven generations 9 7 4 1 2 1 0 1 2 3 4 5 6 7 generation

Violations per generation under a constant gate; the wobble at generation five is fresh-sampling variance, published as part of the data.

Finding 1: Coaching works — until it doesn't

Enriching the guidance eliminated whole violation classes with zero code changes. The authoring rules page — the document the generator reads before planning — started at 354 bytes. Growing it to roughly three kilobytes of concrete rules (declare what you create, verbs must match reality, exact paths, one region per edit) removed entire categories of rejection between generations. Guidance is cheap, fast to iterate, and genuinely effective. If the story ended here, it would be an argument for more coaching.

Finding 2: You cannot out-coach the scaffold

One violation survived two rounds of coaching that named the exact correct answer. The generator kept referencing a file by a lowercase name when the real file is uppercase. We added a rule stating the exact canonical path. It persisted. We added a naming-conventions page spelling out that case is load-bearing. It persisted. The reason: the lowercase reference lived inside the candidate generator's own system prompt — a scaffold of roughly 104 kilobytes. Our coaching, all of it together, was about three kilobytes. Guidance was losing to structure on sheer mass, and no amount of repetition changes that ratio. The fix was one line — in the scaffold, not in the guidance.

Finding 3: The judge is part of the scaffold

The final surviving violation was manufactured by the verification instrument itself. To detect whether a task said "create" or "edit," the gate's input builder lowercased the entire task title — and kept the lowercased remainder as the file path. Every correctly-cased plan was being checked against a case-sensitive file inventory with a corrupted path, and failed. For five consecutive evaluation rounds, the candidates were innocent; the judge was generating the evidence against them. This is the uncomfortable half of the lesson: a verification instrument is itself a contract clause, and it has to be checkable against reality like everything it checks. A judge that corrupts its own evidence fails silently, in the worst possible direction.

What we now believe

Three conclusions, in the order the evidence forced them on us:

  1. Asking is not enforcing. Rules that live in prompts are suggestions; rules that live in schemas and gates are laws. Coaching raises the floor — enforcement holds it.
  2. When correct coaching stops moving a failure, the blocker is structural. Read the generator, not the guidance. Both of our stubborn defects were single lines of scaffold.
  3. Verify the verifier. Gates, checkers, and judges are contract clauses too. The one instrument you exempt from checking is where your system fails silently.

Enforcement did not replace coaching in this run, and coaching could not replace enforcement. The gate held the line while the guidance, the generator, and finally the gate itself were repaired — and the count went to zero and stayed there.

Related writing