# Self-Review Can't Catch a Corrupted Question

> Your verifier approved a clean answer that was still wrong, because it was the right answer to a stale question. Output checks never look upstream.

Published: 2026-08-20 · Canonical: https://agent-coherence.dev/blog/self-review-corrupted-question/

---
An agent which examines its own work is superior to one that doesn't. The inspection only includes what the inspector is able to see, and the question is not visible.

This is a typical example of failure. The support agent prepares a refund decision based on the account record it had read forty seconds earlier. During the time it is reasoning, a peer process causes the customer's plan change to be recorded on that same record. The agent then completes its draft, after which a review stage, carried out by a second model, assesses the draft using the same cached version of the record that the first model had used. The logic is sound, the tone is appropriate, and the figures match up with the record as it was read. It is approved. However, the answer becomes wrong the moment the record is altered, and all the subsequent steps that followed that read agree perfectly with one another.

When faced with this failure the team decided that the verifier needed improvement, whereas in fact the verifier was in good condition and had correctly answered its own question.

The main point is that the verifier checks the answer while no one checks the question. If the input is stale then a correct answer will pass all of the output checks.

## Exactly what does an output check check?

A verifier, a self-review pass, and an eval all examine the same question: does the output result from what the model saw? This is known as output verification and is in fact valuable, the recent work in this area being good work. LangChain released ReviewBench for the evaluation of code-review agents in July and Tuned Evaluators in August, the latter being tuned to perceived error. The aim of all of these is the same: is the output correct?

The gap in the structure lies upstream of the target. If the staleness is due to the agent's input view, a record that has moved, a lossy compaction, or a stale read of shared state, then the output will be internally consistent with the corrupt input. The verifier then carries out its own role, thereby giving the correct answer to the wrong question.

| Layer | The question it answers | What it cannot see |
|---|---|---|
| Eval suite | can this model produce good outputs on known inputs? | the harness that feeds the model in production |
| Verifier, self-review, LLM judge | does this output follow from the view the agent was given? | whether that view still matches reality |
| Input integrity | is the view the agent read still the current version? | whether the reasoning on top of it is any good |

The rows do not lend themselves to being substituted for one another in either direction. Just as surely, a coherent input view coupled with flawed reasoning fails, which is the reason why no argument here is presented against verification. The rows are stacked, and most teams have constructed the top two and have assumed that the third one is also there.

## What prevents a second reviewer from providing help?

The instinctive response to an unreliable agent, namely to include another model to verify it, assumes that the two failures are independent. However, on the input side they are not. The reviewer ends up with the same outdated perspective that the drafter had, or even worse, reads the drafter's output while already having the outmoded assumptions built in. Since both models come after the same corrupted interpretation, their agreement provides information about the reasoning process and not at all about the input. Adding more verifiers increases one's confidence without extending the coverage.

Evals fail to detect it for a simpler reason: they use the weights, and this failure remains within the harness. A dataset that has been purchased can show you that the model reasons well when given the inputs it is provided. It cannot inform you that in actual use the system supplying the inputs had been running a version that had died forty seconds before. The failure escapes detection because the artifact being tested, the model, was not the faulty component.

The version of this process in the coding-fleet deserves to be mentioned since it presents a convincing facade. A reviewer agent checks the difference against the plan, gives its approval and the merge then takes place. During the time the work was being carried out, the plan changed because a peer made a revision that the reviewer had not seen. On the basis of the last write wins rule, the run appears to be green and there is a review in the audit trail which makes it seem intentional. There is a runnable, offline version of precisely this situation in [the v0.13.0 receipt](/blog/shipped-byo-substrate/), in which the merge gate re-triggers the validation when the base has moved rather than relying on the previous approval.

## What closes it

Not a better judge. There should be a check on the input side at the boundary where the agent reads from and writes to the shared state, and this involves a version issue rather than a quality issue. You need to keep track of which version each agent reads. When a peer commits, any agent holding the outmoded view will be detected at its next access, and a write based on that view will be rejected by a typed refusal rather than being carried out. The agent then has to re-read the state and re-derive from it, after which the verifier can assess the reasoning based on the current state. Details regarding [how that mechanism functions](/blog/put-the-version-on-the-write/) and [what each shipped guarantee verifies](/blog/shipped-write-side/) are explained in their own descriptions.

**Scope, stated plainly.** This setup involves a single host and one coordinator, with the writers passing through it. It serves as an input-integrity precondition beneath the verifier and does not take the place of the verifier, nor does it make any assertions regarding hallucinations in the model itself. If reasoning goes wrong even when starting with a completely fresh view, that is within the verifier's domain, and some of the ways in which a fact disappears upstream, such as entering the wrong key, lossy compaction, or a TTL expiry, are application bugs that have [their own levers](/blog/silent-data-loss-taxonomy/) rather than coordination failures. What the input layer is responsible for is more limited and can be checked: the view you based your reasoning on was the current one, or you had been informed otherwise before it was stored anywhere.

## Why it matters

The teams that are putting the greatest effort into verification are those designing genuine agents, and this investment has a drawback which increases as the number of agents grows. The greater the number of agents that share state, the larger the time interval between a read and a write, and that is precisely where the corrupted question exists. A team which has purchased evaluation services, constructed a judge, and included a self-review stage has, by definition, left this aspect entirely uncovered, and each of those individual layers will confidently approve the failure when it arises.

A specific next step is to, whenever a verified output is found to be incorrect, ask one question before making any changes to the prompt or to the model: was the answer indeed correct for the input that the agent actually saw? If the answer is yes, then the fault lies upstream of all the checks you currently have and no amount of improved output checking will be able to catch it. The input-side mechanisms, which include deterministic offline reproductions, can be found at [github.com/Cohexa-ai/agent-coherence](https://github.com/Cohexa-ai/agent-coherence).