Verification · substrate conformance
Apache-2.0 · single host · runs offline · ships the mutants that must failYour agents write to a Postgres row, an S3 object, a file. Somewhere in that path is a claim: that two writers cannot silently overwrite each other. The conformance corpus is how you check that claim instead of believing it — a packaged, runnable kit that certifies the guarantee your binding declares is the guarantee it actually delivers. It ships inside the library, runs offline, and carries deliberate broken implementations that it must reject.
A binding declares a capability tier — a machine-readable statement of the guarantee class it belongs to. The guarantee wording is a closed, tier-keyed table in the library, not free text, so a binding cannot reword its way past the floor. The descriptor validates fail-closed: it may under-claim, never over-claim.
native_cas — the substrate has a conditional write, and the binding rides it.No-lost-update on the version-CAS axis only. A coordinator timeout after a durable substrate write is reconverged by token-identity reconciliation. Single host. A binding at this tier must name its version source.
detect_only — it catches the sequential case and says so.Catches a sequential stale-read then write. Cannot prevent a concurrent race. A binding here passes the corpus by demonstrating the silent loss under a forced interleaving, not by hiding it.
forward_only — ordering, with no undo.Decision-input freshness through invalidation and deny-before-act. The effect itself is forward-only: this layer does not compare-and-swap it, does not undo it, and does not prevent a duplicate. No binding ships at this tier today — the corpus validates the descriptor text, and there is no runtime behavior behind it.
The floor is mechanical: a non-native tier's guarantee text is checked against a forbidden-word list — enforce, cas, rollback, dedup. A detect_only binding cannot describe itself with enforcement vocabulary even if it wants to.
Most of the kit is bookkeeping. Two scenarios are load-bearing, and they are deliberately distinct — because the difference between them is the reason a coordinator exists on top of a substrate that already has compare-and-swap.
A writer that never went through the coordinator moves the substrate underneath you. Your write reaches the substrate's own conditional write and is rejected there: a typed CasVersionConflict, your bytes never land. Postgres and S3 give you this without us. If this is the whole of your problem, use the conditional write you already have.
A coordinated peer commits. Your next act is denied with a typed StaleView before the write is ever attempted — the scenario asserts the substrate CAS call log is empty. The deny being StaleView and not CasVersionConflict is the whole proof: the coordinator layer refused you while you still had time to re-read, rather than the substrate refusing you after you had already spent the work.
Two more legs run alongside them. Never ship a store spies on the coordinator wire and asserts your bytes are not in the payload — only a content hash. Read-pair atomicity asserts one logical read issues exactly one substrate read.
A conformance kit written by the same people who wrote the implementation is worth exactly as much as its mutants. This one ships deliberately broken bindings and asserts that it rejects them. If a mutant ever passes, the kit has lost its teeth and the failure is loud.
The split comparand is the sharpest of them, because it is a bug a version check structurally cannot see. The broken binding returns bytes from one read and a version token from a later read, with a peer's write landing in between. The token is genuinely current. The compare-and-swap passes. The version advances monotonically. Exactly one writer wins. Every property a lost-update check observes is satisfied — and the peer's write is silently gone, because the bytes you derived your change from are older than the token you presented. The loss lives in the pairing of bytes to token, one level below where the check looks.
The corpus runs that control against a conforming binding and requires it to pass, then wraps the split binding and requires the assertion to fire. Two more mutants cover the read-count leg and, in the workspace family, a binding blind to a torn checkpoint — and that one must fail the specific leg it broke, so a failure localizes instead of smearing.
Honest count: three legs have mutants. Invalidation-before-act, the never-ship-a-store wire, and most of the workspace must-match legs do not have one. That is a real gap in the teeth, not a rounding error.
The corpus ships inside the package. There is no vendoring, no copying a test tree, and no dependency on this repository's internals — scenarios describe required final observable outcomes, so a foreign implementation passes by satisfying outcomes, never by mimicking our mechanism.
# Python 3.11+. pytest is optional — see below. pip install "agent-coherence[conformance]"
from ccs.testing.substrate_conformance import ( ConformanceBinding, run_native_cas_conformance, ) # Implement four members over YOUR substrate, then: run_native_cas_conformance(my_binding, make_session)
The seam is a four-member protocol: descriptor (your declared tier), seed, foreign_write (a writer that bypasses the coordinator), and make_view. Implement substrate_read_count too if you want the read-pair leg to actually run — without it that scenario returns early rather than false-greening, and you are simply not certified on it.
Outcomes. There is no CLI and no exit code of its own. A passing run returns None; a failing run raises AssertionError, most of them naming the leg that broke. Your runner supplies the exit code.
Without pytest everything still runs. The one difference is a skipped scenario: with no channel to distinguish "skipped" from "passed," it raises with the install hint instead. It fails closed. It never reports a silent green.
The same scenarios run against a real Postgres and a real S3 bucket, behind a real_substrate marker. Here is the part you would otherwise have to discover yourself:
[coherent-row] or [coherent-object] extra and run the marker.This is the honest shape of it: the default arm proves the logic, and the arm that would prove your substrate is the one you have to run. That is also the engagement — if you want it run against your write path with the result written down, that is a conversation, not a download.
Stated before you ask, because that is the only version of this worth reading.
detect_only binding passes by demonstrating a silent lost update and labelling itself correctly. Never read "conformant" as "prevents lost updates" — read the tier.No, and this is the distinction the corpus exists to enforce. It certifies that your declared tier is honest — that what you claim matches what you do. A binding that declares detect_only passes by demonstrating a silent lost update and labelling itself accordingly. Passing means your label is true, not that lost updates are prevented. Read the tier you declared, not the word "conformance."
No. The corpus imports and runs without pytest; raise-expectations use an internal shim. pytest only changes how a skipped scenario is reported. Without it, a scenario that would skip raises instead, carrying the install hint — it fails closed rather than passing silently, because there is no channel to distinguish skipped from passed.
An in-memory fake substrate that ships in the same module as the scenarios. That is enough to exercise the logic and the negative controls, and it is not evidence about your Postgres or your S3. The real-substrate arm runs the same scenarios against a real backend, and it is not in CI — it runs when a human exports credentials and runs it.
Because they serialize, and a serialized backend false-greens a concurrency test. A test that cannot lose the race cannot prove you would win it. The real arm is gated on a real Postgres or a real S3 bucket for that reason.
The corpus has no CLI and no exit code of its own. A run that passes returns None; a run that fails raises AssertionError, and most failures name the leg that broke. Your runner supplies the exit code — under pytest that is pytest's.
No. Fence-rejection and clock-domain sweep-liveness are deliberately deferred and are not asserted by any scenario. A binding can pass this corpus while claiming nothing about fencing. The gate() effect fence ships in the library and is model-checked in TLA+, but that is a different artifact from this corpus.
15-minute call. Tell me what your agents share and where it lives — a Postgres row, an S3 object, a file, a store key. I will tell you honestly which tier your path is at today, and whether the answer is this library or the conditional write you already have.