What BPOS runs · Judge

Where a person decides.
And why one still must.

Most of BPOS's work commits without anyone looking. Some of it must not. These are the rules that call a person in, the evidence that they are needed, and what a person's decision becomes once it is made.

In BPOS a person is in the loop by rule, not by habit. A person decides when a test failed, when the judgements disagree, when the score is too low, when the platform's own review will not send the work back, or when a sprint asks. A person's decision is then recorded as evidence the platform learns from, never as a setting typed in by hand.

This article belongs to What BPOS runs. The quality gate sets out the mechanics; this article is about the person.

Where a person decides, and why one still must. 14 of 19 jobs a person rejected had already passed the judge, over the last 60 days. A person decides when a test failed its last run, the three judgements disagree, the score is below 0.60, the platform's review will not send it back, or the sprint asks for a person on every job. What the decision becomes: evidence on the paths the work took, never a weight set by hand; a reason the next attempt reads; a question the record must answer, why was it rejected. The judge checks the lines; a person checks that the work still does its job. In 30 days: 21 approvals and 30 rejections, by one person.

Why a person still decides

We took every job a person approved or rejected in the last 60 days, where the judge had scored the work just before the decision. A person approved 21 and rejected 19. Of the 19 rejected, the judge had passed 14. On the work people turned down, our judge was wrong 74% of the time.

The judge was not careless. It did what it was built to do: check each line of the contract against the work. One job it scored a perfect 1.00 was rejected because the table it produced did not measure what the table was for. Every line was matched. The work did not do its job.

The question the judge does not ask

A person asks one more question, one level up: with this change as written, does the larger thing it belongs to still reach its goal? A new function can match every line of its contract and still leave the tool it lives in unable to do its job. That larger goal is read from the record, never guessed: the blueprint a job belongs to, the stage of the program a blueprint serves, the project a program is for. And what worked before the change and belongs to that goal must still work after it.

It is the same test that runs through this series: the one level up of The blast radius of a change, and the question of whether successful paths still reach the goal in A goal has necessary stages. Today's judge never asks it. That is why a person still does.

When a person is asked

  • A test failed its last run. The work never commits on its own.
  • The three judgements disagree by more than a limit set in the platform's graph.
  • The score is below 0.60.
  • The platform's own review will not send it back. Work routed to review is first read by the platform's failure judge, which sends back what it is confident about while the job's attempts are converging. What it will not send back goes to a person, with its diagnosis already written.
  • The sprint asks for a person on every job, whatever the score. In the 30 days to 9 October, 86 of 268 sprints did.

Who is asked is declared too, in order: the person the job is assigned to, then the sprint's, then the last person who acted on it through the connector, then the project's owner.

What a decision becomes

A decision is not the end of the matter. It becomes three things.

  • Evidence. A person's verdict enters the record as an observation on the paths the work took, through the same rule as any outcome. Nobody sets a weight by hand, a person included; Nobody Types the Weights explains why.
  • A reason. The next attempt reads why the last one was rejected.
  • A question. Since September, a rejection is something the platform must explain: the round that asks why each job ended where it did now has to account for a person's no, not just pass it on. The Gettier problem for AI agents describes that round.

The reasons add up. When we read 113 of them, only 5% cited a compile or type error, and at least 29% named something else the change should have touched. That is where the blast-radius work came from.

The judge, rebuilt

Two decisions follow from the 74%. First, every judgement reads one constant pair of models, whatever pair the work itself runs on. If the work moves to a larger model, the ruler does not move with it; changing the ruler is a recorded calibration, never a side effect. A score has to mean the same thing from one job to the next.

Second, the judge is being rebuilt as a procedure instead of a checklist: read the contract, ask the parent's question, decide what to read, then weigh the evidence. Its reasoning is recorded step by step, so a false pass shows where the ruler went wrong. The reasoning is to go to the round that learns from it, and the next attempt is to get the verdict and its grounds. A test of the rebuilt judge against decisions people have already made is running now. We will publish what it finds once we have read it.

Where it falls short

  • "A person" is one person. In the 30 days to 12 October there were 21 approvals and 30 rejections, all by the founder. The rules do not depend on who decides; the record would be stronger with more than one.
  • The judge in production still checks lines. Until the rebuilt judge is proven, the parent's question is asked only by a person.
  • The judge's own findings are not yet learned from. What it found per criterion is stored, but no loop reads it yet; its reasoning will be the first part that does.

Common questions

What does human in the loop mean in BPOS?

A person decides by rule, not by habit: when a test failed, when the three judgements disagree, when the score is below 0.60, when the platform's own review will not send the work back, or when a sprint asks for a person on every job. Everything else commits on its own.

Why can't the AI judge decide alone?

Because it checks the lines of the contract, not whether the work still does its job. In the last 60 days it had passed 14 of the 19 jobs a person then rejected. The question it does not ask is whether, with this change, the larger goal the work serves is still reached.

What happens to a person's rejection?

It becomes three things: evidence on the paths the work took, entered through the same rule as any outcome and never as a weight set by hand; a reason the next attempt reads; and a question the platform's record must answer about why the work was rejected.

Sources

  • Our own records: people's approve and reject decisions over the last 60 days, matched to the judge's score stored just before; decisions over the 30 days to 12 October 2026; sprints asking for a person over the 30 days to 9 October; 113 rejection reasons read in September 2026.

Related writing