Evidence
The concepts we run on —
with the evidence and the caveats.
Each concept below gets a definition you can lift, the independent work that arrives at the same conclusion, what we actually run in production, and the honest limitation — on the page, not in a footnote. None of these authors know our stack: this is convergence, never endorsement. Only entries our register grades as converging render here.
The scaffold
Definition. The fixed structure around a language model that determines what its output is allowed to become: the schemas it must fill, the gates its work must pass, the verifiers that judge it, and the delivery machinery that carries knowledge to it. The scaffold is the part of an AI system that cannot be talked out of its job.
Independent corroboration
- The Model Wasn't the Bottleneck. The Configuration Was. — Akimitsu Takeuchi, AI Advances (Medium), 2026-06.
The title is the thesis: execution quality moved with configuration, not with model choice — independent practitioner arrival at the same dominance ordering we operate on.
Converges (hard) — the title is the thesis. - Elon Musk Just Did to Coding What SpaceX Did to Rockets — Jaroslaw Wasowski, Level Up Coding (Medium), 2026-06.
The surrounding harness, not the model, is credited as what moves an agent's capability — the scaffold framed as the reusable rocket: the engineering that makes the expensive part productive.
Converges on configuration-over-model.
What we run
Our delivery platform treats the model as the smallest part: plans are validated by mechanical gates before execution, executions are graded against acceptance criteria with verbatim-quote verification, and no instruction depends on the model choosing to comply. When behaviour must change, we change the scaffold, not the prompt.
Caveat. The external pieces below argue the position from practice and reporting rather than controlled study; our own strongest evidence is the configuration-dominance pattern across production executions, which is observational within one platform.
Deepest treatment: You Cannot Out-Coach the Scaffold.
Knowledge arrives at execution time
Definition. Knowledge helps an agent when it is delivered as context at the moment of execution, selected for the task at hand — not pre-loaded as instructions the agent is asked to remember.
Independent corroboration
- The Physics of Multi-Turn Long-Horizon Planning — Men, Jin, Liu, Zhao — CAS Institute of Automation, arXiv:2607.24720, 2026-07-27.
In a fully controlled synthetic environment, injecting task knowledge as context at execution time causally outperforms training it into weights; their ablation attributes the largest planning gain to initial context, not mid-flight adaptation.
Converges (hard) — controlled experiment, model held constant. - How to Build a Self-Improving AI Harness — Lilian Weng, lilianweng.github.io, 2026-07-04.
A reference architecture for agent improvement built on verifier-grounded failure records delivered back into the working loop — knowledge as operational context, not as retrained instinct.
Converges (hard) — the field's leading practitioners describe the loop we run. - Our own null result: 88-execution checklist sweep — Bouletteproof (internal, published), bouletteproof.com, 2026-06.
Arm-randomised across 88 production executions: skill checklists as prompt tokens produced no measurable quality lift. The same knowledge, delivered per-task at execution time, is the mechanism we kept.
Our own published result — published including the null.
What we run
Our platform delivers capability pages to each working agent over HTTP at execution time, selected per task. The alternative — the same knowledge pasted into prompts as checklists — was tested in an 88-execution arm-randomised sweep and produced no measurable quality lift; we deleted it and published the null.
Caveat. The external studies below operate on training-side substrates we deliberately do not touch; mechanisms converge, magnitudes do not transfer. Convergence is not endorsement — none of these authors know our stack.
Deepest treatment: The Checklists That Did Nothing.
The verifier decides, not the narrator
Definition. In a trustworthy agent system, outcomes are decided by an external verifier the agent cannot rewrite — a compiler, a sanitizer, a mechanical gate — never by the agent's own account of its work.
Independent corroboration
- Building an Agentic Security Pipeline That Finds, Proves, and Patches Vulnerabilities — Fareed Khan, AI Advances (Medium), 2026-06.
A six-step defender loop whose core lever is refusing to trust the agent's account: the host re-runs the proof-of-concept, and a crash — not the model — decides what counts.
Converges (hard) on the verifier-decides gate. - MANTA: inference-time topology adaptation — Huang, Wang, Lai et al. — Cornell / UIUC / Academia Sinica, arXiv:2607.28527, 2026-07-30.
Their trace auditor requires verbatim evidence quotes checked against the cited trace — one invented reference or quote rejects the finding. Verification restructured into the output contract, where checking is mechanical.
Converges (hard) on constraint content; their evidence supports fixed-architecture constraints.
What we run
Every plan our generator produces passes a mechanical grounding gate before acceptance; every execution is graded against acceptance criteria with evidence and locators; and as of August 2026 the grader itself must cite an exact verbatim quote from the produced files for every passed criterion, string-checked against the bytes.
Caveat. The strongest external source below comes from a domain (C memory safety) where a sanitizer provides objective ground truth; our gates approximate that objectivity elsewhere and we say so. Verifiers are contract clauses too — the one you exempt from checking is where your system fails silently.
Deepest treatment: You Cannot Out-Coach the Scaffold.
Learning-loop hygiene
Definition. A system that learns from its own outcomes must protect the loop itself: criteria frozen before data exists, feedback restricted to sourced facts, and bad trajectories kept out — because a wrong label in the loop induces regressions instead of improvement.
Independent corroboration
- The Physics of Multi-Turn Long-Horizon Planning — Men, Jin, Liu, Zhao — CAS Institute of Automation, arXiv:2607.24720, 2026-07-27.
Suboptimal trajectories in training data catastrophically collapse long-horizon planning — errors compound; hygiene of what enters the loop dominates outcomes.
Converges (hard), causally identified in a controlled environment. - MANTA: inference-time topology adaptation — Huang, Wang, Lai et al. — Cornell / UIUC / Academia Sinica, arXiv:2607.28527, 2026-07-30.
Their published self-audit — auditor precision 37.5% as an incorrectness proxy, one in five repairs increasing flag count — demonstrates why unverified judgments must not drive automatic change; bounded, gated mutation is their answer and ours.
Converges on gating discipline; honest about their own auditor's limits. - Our own instrument record: the four-run hysteresis backtest — Bouletteproof (internal, published), bouletteproof.com, 2026-08-01.
Two instrument defects and one mis-implemented criterion found across three failing runs — each fixed in the machine while the pre-registered criteria never moved; the fourth run passed all three.
Our own published result — failures published as engineering.
What we run
Verdict criteria are pre-registered before any arm data is examined, and a criterion found wrong mid-window restarts the window rather than moving the line. Rejection feedback to agents carries only sourced facts — compiler output, log lines, query results — never theories. Our page-health state machine cleared a four-run backtest whose criteria never moved: fail, fail, fail, pass.
Caveat. The external causal evidence comes from a 100M-parameter synthetic environment: mechanisms are trustworthy, magnitudes are not transferable. Our internal evidence (wrong-theory feedback inducing regressions) is observational within one platform.
Deepest treatment: How We Do SEO: Signals, Not Schedules.
Our own result, published with the bad parts intact: HikrLink, built by the loop — 122 quality-scored executions across four repositories, including the two newest, which score worst.
Register updated 2026-08-01. Entries carry their original verdicts from our internal External Validation Register; banner-grade and noise-grade material never reaches this page.