Practical guide
Investigate where two implementations disagree
During a migration, two implementations can run successfully and return different answers. Differential testing applies the same input to both and compares their observable behavior.
Decide what equivalence means
Byte equality is appropriate for some pure transforms. For other interfaces, timestamps, map order or generated identifiers need a reviewed normalization. Error behavior and side effects can be part of the contract too. Choose the comparison before running the campaign; a normalization that removes the meaningful difference can make a broken replacement appear equivalent.
Illustrative example: URL normalization
| Input | Implementation A | Implementation B |
|---|---|---|
| A URL with repeated query keys | Preserves both values | Keeps only the last value |
Both outputs may be syntactically valid. The disagreement does not identify the correct implementation. Read the approved contract and callers: do repeated keys represent a list, an error or last-value precedence? If the old implementation has a known bug, copying its output would preserve the bug rather than satisfy intent.
Use a controlled comparison harness
Feed the same language-neutral input to both sides. Record their versions, environment, outputs and failures. Separate “both ran and disagreed” from a compiler error, missing dependency or timed-out shim. Minimize a divergence while preserving the mismatch, then replay it outside the search loop. Agreement only establishes parity for the compared observations and tested inputs.
Proof's current differential fuzz path
The current CLI supports a configured pure transform with Python and Rust shims under project.checks.differential_conformance.transforms. It mutates existing proof.fixtures/v1 cases, compares both shims and can shrink real divergences into unpromoted candidate known issues.
proof differential fuzz normalize_url --seed 7 --iterations 200
The transform key is illustrative and must already exist in project configuration with both shims. The seed makes generated input selection replayable; reproducing outputs still requires stable implementations and environment. This command is not a universal comparator for arbitrary deployments.
Turn the mismatch into a reviewed requirement decision
Inspect the candidate, decide which behavior is intended and assign severity based on actual reach and impact. Candidates are not automatically promoted into accepted findings. Keep the minimized input as a regression and link it to the corrected requirement or known issue. If both implementations share the same defect, differential agreement will miss it; independent properties and expected examples remain necessary.
Follow the evidence
Choose one pure transform, define its comparison contract and retain the first replayable divergence.
Review one implementation divergence. Bring the minimized input and both outputs so we can compare them with intended behavior and identify the evidence needed for a migration decision.
Add an independent oracle · Preserve behavior during migration