A hand-written map
can be worse than no map.

A procedural graph holds a task’s know-how outside a model, as steps and the moves allowed between them. It can be read and corrected without retraining anything.

If you have written down how your agents should work, in runbooks, pages or prompts, this paper asks a plain question. Is that knowledge helping them, or getting in their way?

The paper is by researchers at Google, Georgia Tech and Peking University, published in September as arXiv:2609.09153. We build the same kind of object, so its answer applies to us too.

What repaired the score

Following the expert map was worse than having none. One offline revision of the map made it worse again, at 53.57.

The score recovered only when the map could change during use. Each change was kept only if a held-out test did not fall. That reached 92.86.

A map built from nothing, under the same loop, reached 91.07. On a second benchmark, the map built from nothing was the best result of all.

The value was never in the map. It was in the loop that corrects it. A map without that loop is a liability that sounds authoritative.

Where that leaves us

Our knowledge pages are hand-written. The part of our system that proposes corrections to them runs at volume.

As of 22 September, it had rejected 1,072 proposals, held 291 for observation, and promoted 22, all on a single day.

The part that measures whether a correction helped has not kept up. Measured outcomes exist for six proposals.

In the paper’s terms, our correction loop is open. Until it closes, our hand-written pages are a hypothesis, not an asset.

Each has the half the other lacks

The paper’s changes arrive in batches. A reviser compares failed runs with successful ones and proposes edits.

A batch is kept only if a held-out score does not fall. Rejected batches are remembered, so a bad idea is not proposed twice.

Our changes are continuous. Every link carries a weight that moves with the results of the work that used it. Nobody sets that weight by hand.

They cannot tell which link earned an improvement. We cannot tell whether a promotion made things worse. We are adding their check to our system.

Three findings worth taking

Less of the map beats more of it. Giving the agent the whole map dropped one benchmark score from 72.58 to 54.48, at 96,360 tokens per task. Showing only the two steps around its position scored 81.53, at 28,064 tokens.

Faster is not cheaper. Local guidance cut the agent’s steps from 28.20 to 18.57. Total token use still rose 33.4%.

Report what you keep, not the best you saw. Their final map scores 85.0 on test. One search round scored 95.0, and they do not headline it. Quoting it would mean choosing on the test set.

They also warn that with twenty test runs per split, single keep or reject decisions turn on one or two runs. We now say the same about our own replays.

What we are taking, and what we are not

Taking: their check on structural change. A promotion lands only if a held-out measure holds.

Taking: their rejection memory. A refused proposal becomes evidence, not a repeat.

Not taking: revising a link by deleting it and adding it back. That throws away everything the link has built up. A reworded description is not a new link.

The history on a link is the one part of the record we cannot rebuild.

Common questions