Topic · differential testing

What is differential testing, and how do I keep the two implementations true in CI?

Gist

Differential testing feeds the same input to two implementations and compares the outputs. Proof hunts a new disagreement on one declared transform. proof differential fuzz vertex_ocr.request mutates the fixture corpus, runs both shims, and shrinks a real divergence to a replayable case. A mirror ledger is not this. Hypothesis still samples one program. Jama still authors.

proof differential fuzz vertex_ocr.request

Keep the golden master if you pin observed bytes. Keep Jama if you author the programme. Proof is not a hypervisor, and a green corpus is not two-solver agreement.

Request demo Last golden master vs this graph

01 · The input that both sides see

A cell on both sides is not the same output.

A coverage ledger can prove each implementation has a test for the same class. It cannot prove they agree on one concrete input.

The rewrite-with-a-ledger H1 lives on characterization testing and mirrors. The fixture-from-YAML H1 lives on property based testing. This page is the parity check: one language-neutral input, two shims, compare.

Register both entrypoints under project.checks.differential_conformance.transforms. A transform with one shim is skipped as information, so you adopt one key at a time. The fuzz run is deterministic. The same --seed reproduces the same sequence. Candidates are never auto-promoted. A person grades severity and moves a finding into the curated corpus.

proof differential fuzz vertex_ocr.request
proof audit --check differential_conformance
proof audit --fail-level warn

The first command hunts a new disagreement. The second replays the authored fixture corpus. Pasting last week's golden bytes into both trees is how two wrong answers stayed friends.

02 · The exhibit

Same input. Two answers, or one.

The golden master still matches each side's own suite. The fixture will take the input both shims see. Click the tabs.

Python shim / Rust shim

Opened each side's own golden file
input never compared across the pair

Both suites green

This push

One side converts the URL. The other passes it through.

No shared input

Python shim / Rust shim

Still last golden. Still two suites.

Keep the shims

Proof

Ask same input, same envelope
Gate differential_conformance, then fuzz

diverge-cross-impl

Same input. Two answers, or one. Click the tabs.

Surface	What they do	What Proof does
Differential testing	Same input to independent implementations; a mismatch is a bug in at least one	`proof differential fuzz` on a declared transform. Corpus replay is `proof audit --check differential_conformance`.
Characterization / mirrors	Pin observed bytes, or prove the same evidence cell exists on both sides of a rewrite	That H1 lives on characterization testing and mirrors. A complete mirror is not identical output.
Property based testing	Random generators and shrinking against one program	That H1 lives on property based testing as `proof proptest`. Fixtures are not two shims.
Mutation testing	Kill mutants to see whether the suite notices	Not this product. PITest stays with that SERP. The class-of-bugs H1 lives on bugs my tests never check.
Antithesis / whole-system simulation	Deterministic schedule over a hypervisor or a simulator	Not this product. Two shims on stdin are not a cluster schedule.
Z3 and cvc5 on one lemma	Two solvers, one formula	That is `proof verify-lemma --solver z3,cvc5` on Z3 / Kind2. Solver disagreement is not a shim disagreement.

A fixture names one requirement and one transform key. Each case carries an input and an expected tagged union: value, error, or any. Both shims must succeed with canonical-JSON-equal output, or both must refuse. Exactly one success is diverge-cross-impl.

project:
  checks:
    differential_conformance:
      fixtures_dir: proof/fixtures
      transforms:
        vertex_ocr.request:
          python:
            bin: .venv/bin/python
            module: proof.shims.vertex_ocr_shim
            function: request
          rust:
            command: cargo run -q -p litellm-core --example proof_shim -- vertex_ocr.request
            dir: litellm-rust

proof differential fuzz vertex_ocr.request
proof audit --check differential_conformance
proof audit --fail-level warn

A half-registered transform is skipped as info. Treat a candidate KnownIssue as an ungraded finding, not as a merge block, until a person promotes it. We have not run this against a frozen Csmith / Antithesis corpus, and we have not claimed the fuzz proves the Go. The loss is named, not scored.

03 · The honest loss

A green differential run is not two programs being equal.

Proof compares envelopes on a declared transform. It does not invent the second implementation. Jama still authors. Hypothesis still samples.

Proof does not write the Python shim or the Rust shim for you. A transform with fewer than two complete entrypoints is skipped. Candidates are never auto-promoted. Fuzz hunts new cases; it does not replace the authored corpus. Temporal order is a different command: proof differential monitor over proof.traces/v1, not a byte compare. Object-code coverage on C still wants the qualified toolchain. That H1 lives on MC/DC for Go and on Proof vs LDRA.

The rewrite-with-a-ledger H1 stays on characterization testing and mirrors. The one-implementation fixture H1 stays on property based testing. Jama still authors.

Request demo How the install is bought

04 · Nearby questions

What people type next.

What is differential fuzzing? Same cluster, shorter head. Same command. Not a twin.

How do I verify a rewrite behaves identically to the old system? Identical output is this URL. The ledger H1 stays on characterization testing and mirrors.

What is property based testing? Fixtures from authored variable semantics, one program. property based testing.

How do I prove a specific function meets its specification using something like Z3 or Kind2? Two solvers, one lemma, not two shims. Z3 / Kind2.

How do I find the class of bugs my tests never check for? Realizability and Kind2 on the shall. that class.

Is Proof a VectorCAST alternative? No. LDRA owns that cluster. Proof vs LDRA.