Evidence & Null Results
Limitations,
perceived and real.
An AI assistant with full access to our code told us our system could not do eight things. It could do seven of them. The one real limit made no noise at all.
A perceived limitation is a capability a system has, which someone concludes it lacks, from evidence the system itself produced. If an AI assistant works on your systems, ask one question before building any workaround it suggests. Is the limit real? In our case, it was real one time in eight.
Eight claims, one session
The claims came from a single working session on 12 September. Each one started from a real artifact. The assistant read it and drew the obvious conclusion.
Seven claims were triggered by something visible, and all seven were false. The only real limit left no trace.
The signal is too sparse. A coverage figure looked tiny. It was computed from two columns nothing had written to since July.
The ranking has no table. A table with the right name was empty. The ranking writes to another table, which holds over 144,000 weighted links.
The new rule must replace the old ranking. Two components shared one name. The rule feeds the one that walks between pages, not the one that ranks jobs.
The system never explores. It seemed to follow chosen links only. At each single step, it follows every written link, up to a budget.
A safety check cut off part of the knowledge base. The check blocked one way of deriving it. Looking a page up by name was never affected.
A popular page can never lose rank. Every update observed was positive. A decay applies on every update, so scores can and do fall.
One of our web file formats cannot be written. No special handling was visible. A comment in the source says the format was left off a protection list on purpose.
The ranking saturates. Nothing suggested this one. It was the only claim that was true.
Every false claim had an artifact behind it. Believing one costs a sentence. Checking it costs an afternoon.
The one that was real
Every link in our knowledge base carries a score. It rises when work that used the link goes well, and fades when that work goes badly. The ceiling is 0.95.
The update rule added new results one and a half times as strongly as it let old ones fade. So a link’s score settled at one and a half times the quality of the work using it.
Any link used by work scoring 0.633 or better hit the ceiling and stayed there. From 1 August to 12 September, 74.6% of 3,946 scored runs cleared that bar. At least half of them scored a perfect 1.0.
Above 0.633, every link ended at the same score. The shaded band is where three quarters of the work landed.
For three quarters of the traffic, the ranking could no longer tell good paths from bad ones. Nothing flagged it. The code ran correctly and wrote correct values.
The fault lived in how two settings related to each other. We fixed it the same morning by setting them equal. A link’s score now settles at the average quality of the work that used it.
A setting can be right on its own and wrong in combination. The only way to see it is to work out where the numbers settle.
Why the loud ones are usually wrong
A limit that throws an error is one somebody already found. Errors are written by people, for cases they expected.
A limit nobody has found emits nothing. Nothing in the system knows it is there to announce. So a working system shows far more apparent limits than real ones, and the apparent ones are louder.
All seven false claims share one shape: empty table, therefore missing mechanism. It is a pattern match made under uncertainty, from a genuine observation. Care does not prevent it. Checking does.
We made the same mistake writing this
The first draft of this article listed the wrong seven. It was built from a summary of the session, not from the audit written during it. Three of its examples came from other conversations.
We caught it when we went back to the primary record to register the claim. Same error, one level up: a summary read as the source. The fix was the same too. We read the original.
What we changed
One rule now applies to every assistant that works on our system. A claim that the system cannot do something needs a citation first: the file and the line.
An error message is a symptom, not an architecture. If no citation exists, the honest answer is “I have not read that yet”. Then read it.
The rule costs one search, and it runs before any workaround is designed.
In this session, skipping that search would have scoped seven pieces of work against limits that did not exist.
Common questions
What is a perceived limitation?
A capability a system has, which someone concludes it lacks, from evidence the system itself produced.
How do you tell one from a real limitation?
Ask what produced the evidence. Empty tables, errors and unresolved names were all written or left by someone. A limit nobody has met yet has no author, so it leaves no artifact.
Why would a real fault produce no error?
Because correct and useful are different things. Code that correctly writes a badly defined number will run cleanly forever.
Is one session a sound sample?
No. This is a self-audit of one session, reported because the ratio was striking. The reasoning error is the part that transfers, not the count.
Related writing
- A Green Build Is Not Knowledge. Why passing is not the same as verified.
- We Deleted 20 of Our Own Quality Checks. The audit that separated checks from ground truth.
- The Model Is the Smallest Part. Why the system around the model carries the quality.
- LLM-as-a-Judge: Perfect Scores 92% of the Time. another instrument that was easy to read too kindly.
- Don't Build Your Safety on a Chain of Thought You Don't Own. why an assistant’s reasoning is evidence to check, not a result.