Proof · Engineering practice
Why do tests pass while bugs still ship?
Tests can pass while bugs remain because the suite asks only the questions encoded in its inputs and assertions. A missing boundary, an incorrect expected result or an untested integration can leave an approved requirement violated.
Three passing cases miss the lower bound
This small Python example was executed for this article on 2 October 2026. It is a teaching reproducer, not a product incident. The requirement is to clamp the worker count to 1..20; the implementation clamps only the upper bound.
def workers(requested):
return min(requested, 20)
for requested in (1, 10, 30):
assert workers(requested) == max(1, min(requested, 20))
print("Existing cases: 3 passed")
actual = workers(0)
print(f"Boundary case: requested=0, expected=1, actual={actual}")
assert actual == 1
The first three assertions pass. The added assertion fails: requested=0, expected=1, actual=0. It identifies a concrete violation of the stated lower bound. Replacing the implementation with max(1, min(requested, 20)) addresses this example; the actual product may also need type, overflow and resource-limit cases.
A model counterexample answers a different question
A realizability check asks whether some implementation could satisfy the modeled guarantees under the declared assumptions. If the contract requires both at least one worker and no more workers than requested, an allowed request of zero makes those guarantees incompatible.
proof validate --preflight specs/system evaluation
proof realize specs/system evaluation --diagnose --persist=false
These commands assume the project contains a system specification and a component named evaluation. Preflight checks its model; realizability invokes the solver on the generated contract. A diagnostic counterexample concerns that model. It is not automatically a failing execution of the helper, nor evidence that a whole implementation defect class has been found or closed.
Keep the failing case and inspect its neighbors
Add the missing behavior to the evidence plan, preserve the approved lower-bound requirement, and record the reproducer before making the fix. Run the original case after the patch, then examine related inputs and dependent callers. Keep the reproducer as permanent regression protection.
| Claim | Evidence needed |
|---|---|
| This instance is fixed | The same failing input now produces the required result on the fixing revision. |
| The requirement was reverified | The declared boundary, nominal and failure-path checks ran successfully. |
| The defect class is closed | The declared class scope and broader obligations are established, with assumptions and remaining exceptions explicit. |
Mutation and property testing can help discover gaps before a production incident. They need assertions derived from intended behavior, not just a high aggregate score.
Run the checks and preserve their scope
proof audit --check tests_pass
proof audit --check obligation_evidence_complete --verbose
proof audit --fail-level warn
Test execution, obligation-evidence structure and the overall audit are separate checks. The effective fail level controls the CLI exit; merge enforcement also requires provider rules, revision binding and restricted bypasses. A green audit describes the checks and scope selected, not every possible production input.
For a real published limitation, read the two jsonparser misses. For the denominator question, see test coverage versus requirements coverage.