A green build is
not knowledge.

A green build is a compilation event, not a verified claim. Verification means a pass or fail check against what was actually asked, with a reason you can point to. That is the definition we hold ourselves to, and it took us an uncomfortable amount of failure to arrive at it.

What green actually asserts

When a pipeline goes green, three things have been established: the code parses, the code compiles, and the tests that someone previously wrote still pass. That is all. Green does not assert that the change does what the ticket asked. It does not assert that the tests cover the change. It does not assert that anything was learned.

This was always true, but human review papered over it. A senior engineer reading a diff carries the intent in their head and checks the change against it. Remove the human, let a model write the change, and the pipeline is suddenly the only reviewer left standing. It was never built for that job.

Where it bites: three failure shapes

The test that cannot fail. A check whose failure mode is silent is not a check. We found verifications in our own system that piped their output through a truncating command and read the exit code of the wrong process. They reported success on every run, including the runs where the thing they verified was broken. The line was green because the thermometer was painted on.

The check that passes for the wrong reason. Output can satisfy a test by accident: a stub that returns the expected constant, an error swallowed before the assertion, a fixture that happens to match. The build is green. Nothing true has been established.

The pass with no why. A model produces working code and the pipeline confirms it works. If nobody asks why it worked, the next run cannot reuse the success and cannot avoid repeating the failure that preceded it. A green build without a recorded reason is a coin that happened to land heads.

What we do instead

We run AI-written changes through an acceptance step before anything ships: pass or fail checks on whether the output does what was asked, not just whether it compiles. Behind that sits a quality score on every execution, and behind that a rule we learned the hard way.

Rejection feedback carries only sourced facts. Compiler output, log lines, query results. Never a theory about what went wrong. We proved on our own system that an unverified causal theory placed in review feedback makes the next attempt worse, because the model obeys the theory instead of the evidence. Theories stay in discussion, labelled as theories. The full incident is written up in a companion case study.

Every verification must be able to fail loudly. A check is only admitted if we can demonstrate the failure path fires. Checks that stop the line without checking are worse than the faults they were built to catch.

After a pass, we ask why. Each completed run gets an explanation step: what worked, what the evidence for that is, and what should be reused. The pass becomes a claim with a source instead of an event with a colour.

The volume matters here. Across 12,103 recorded gate decisions and 331 sourced rejection notes on our own platform, the pattern held: the runs that produced reusable knowledge were the runs where the reason was captured, not the runs that merely passed.

The uncomfortable part

We are not writing this from above the failure. The silent verifications were ours. The painted thermometer was ours. The point of publishing it is that the fix is structural, not motivational: you do not get verified knowledge by caring more, you get it by making "does what was asked" a gate and "why it worked" a required artifact.

A green build is a good morning. It is not knowledge until it can say what it knows and how.

Common questions