Mutation testing: can your tests detect a broken requirement?

Ask whether an assertion notices a relevant fault

Mutation testing changes an implementation and observes whether its tests fail. A mutant is killed when a test fails with the change active; it survives when the tests still pass. The result helps investigate the assertions, but it does not decide which behavior the product should have.

This example executes two hand-applied mutations in a short token-access function. It uses an ordinary Python runner, not Stryker, an automatic mutation engine or a Proof audit. The results below are actual local executions of a synthetic teaching fixture.

# Baseline
return active and now < expires_at

# M1: allow the expiry instant
return active and now <= expires_at

# M2: remove the revocation check
return now < expires_at

Inspect all three implementations. The weak suite tests only an active token immediately before expiry. That input returns true in every variant, so the assertion cannot distinguish either fault.

Justify the new assertions from the requirement

The fixture's authored contract, EX-TOKEN-1, permits access only while the token is active and strictly before expiry. Its fictional service owner defines expiry as an exclusive upper bound. That independently states the expected behavior; the runner does not copy its oracle from the baseline or the mutated output.

• Active token, time 99, expiry 100: allow.
• Active token, time 100, expiry 100: deny.
• Inactive token, time 99, expiry 100: deny.

The three input cases express those clauses directly. If a real product instead promises validity through the expiry instant, the first mutation may implement the intended behavior. Resolve that authority question before strengthening a test. Adding an assertion solely to improve a mutation score can preserve the wrong requirement.

Reproduce the experiment

Download the source and captured results, extract the archive, then choose a new output directory:

cd evidence-methods
python3 run.py --output rerun-results

The runner uses Python 3.9 or later with its sqlite3 module. It needs no pip packages or network access and refuses to overwrite retained results. Exit zero means the teaching outcomes match the expected contrast, including the deliberately rejected migration candidate.

Captured October 3, 2026 at 09:30:30 UTC with Python 3.14.7 and SQLite 3.53.4. The result includes source/input hashes, command, runtime and individual observations. Result SHA-256: 8819dd948702332953a3cf281fe8b70ec7f8a4fa81f2509cda04a9a166596f92. Hashes identify these bytes; they are not an independent signature or proof of origin.

Keep the denominator and the remaining gaps visible

This run contains six implementation/suite combinations and twelve individual assertion observations. The stronger suite kills the two selected mutants. That count says nothing about ungenerated mutations, other functions, invalid inputs, concurrent revocation or clock behavior outside the injected integer-time model.

A survivor deserves investigation: the code may not have been exercised, the assertion may be weak, or the mutation may preserve the behavior relevant to the requirement. An equivalent mutation cannot be distinguished by a behavioral test in that scope. A real mutation campaign should also account for invalid mutants, timeouts, exclusions and the baseline test result. Avoid presenting two selected kills as a general test-quality percentage.

Use the result to improve a justified assertion and retain a regression check. In Proof, connect that evidence to the relevant requirement and issue record, preserving which source revision and cases were tested. This example does not demonstrate an automatic mutation-report importer or establish defect-class closure.

Method source

Stryker's official mutant states and metrics, checked October 3, 2026, supplies the killed/survived terminology and distinguishes other outcomes. Stryker was not run here. The linked source files and captured observations are the evidence for this teaching example.