Topic · Characterization testing

How do I rewrite a legacy system without introducing regressions?

Proof treats the old tree as source and the rewrite as target. The audit fails if the new tree dropped a test, an MC/DC row, or a known-issue reproducer the old tree still had.

proof audit --check mirror_complete

Characterization tests pin observed output. They do not notice a guard that never made the jump. A thinner suite on the new code still goes green.

Request demo The campaign

01 · What characterization testing is

Lock what the old system does, then change the code.

Michael Feathers called it characterization testing: write tests against the behavior you observe, not the behavior you wish you had, so a rewrite cannot quietly drift. Golden-master testing is the same idea with captured output. ISO 26262 back-to-back testing runs the same inputs through a reference (the old version, or a model) and the new implementation, and compares the results.

That oracle is real work. It catches an implementation that no longer matches the old bytes. It does not catch the other failure of a port: the feature looks done, the new tests pass, and the malformed-input rejection, the MC/DC row, or the Reproduces: KI-x guard never made the jump. Nothing fails, because the new code is graded against its own thinner suite. The loss is invisible until the old bug comes back.

Proof's answer is not another output dump. It is a mirror: a contract that the target file-set must reproduce the verification ledger of the source, for every requirement in scope. Coverage parity, keyed by cell, not by test name.

02 · The exhibit

A live Python to Rust port. Eleven findings still on the board.

Mistral OCR, LiteLLM. Source is the authority. The campaign splits what the target inherited, what it introduced, what diverged, and what it already fixed. A cell the target dropped still fails the mirror.

Proof campaign MIR-001, Mistral OCR Python to Rust, in progress. 11 findings, worst Medium. Source litellm/llms/mistral/ocr/transformation.py; target litellm-rust crates. Findings grouped: the target's own issue (2), inherited from the source (5), diverges on the target (1), fixed in the target (3). — MIR-001. Inherited source issues are not "the rewrite broke it." A missing cell on the target is.

03 · The command

Declare the mirror. The audit is the burndown.

Source is the authority. Target must be a superset on each enabled dimension. Extra target cells never fail. Status in_progress tracks gaps as warnings. Status complete turns remaining gaps into errors that gate the audit.

proof mirror add --id MIR-001 \
  --source "litellm/llms/mistral/ocr/**" \
  --target "litellm-rust/crates/core/src/providers/mistral/ocr/**" \
  --status in_progress

proof audit --check mirror_complete
proof mirror status MIR-001

A Python test and a Rust test count as the same cell if they witness the same requirement and discriminant. The names can change. The cell cannot vanish.

Request demo Instruments

04 · The honest loss

A complete mirror is not identical output.

Mirror proves the refactor reproduced the same verification obligations and coverage: no dropped obligation-evidence cell, no dropped regression guard, no dropped MC/DC row. It does not, by itself, prove the two implementations produce identical output. Both sides can carry a malformed_input:negative cell and still reject different inputs, because each test was written against its own implementation.

Characterization tests have the opposite limit. They pin the old behavior, including bugs you meant to drop. A golden master that records a wrong answer will fail the rewrite that finally corrects it. You still need a requirement to say which behaviors were intended and which were incidental. A mirror of an under-covered source is meaningless; it only copies the holes.

True input-output agreement is a different tool (proof differential fuzz on a declared transform). It is not what mirror_complete checks, and it is not a characterization suite for a whole legacy system. If the rewrite has no approved requirements yet, recover them and have the owners sign them before you treat either ledger as the bar. The traceability matrix is the other direction of the same problem: links that are asserted once and never re-read.