go test / PITest
- Looked at this input, this mutant
- The pair not in scope
Topic · Untested class
How do I find the class of bugs my tests never check for?
Gist
The suite is green on the inputs you wrote. Mutation testing still scores those tests. Proof asks whether the shalls can be kept at all. Kind2 returns a counterexample when they cannot, with the same command on every push.
proof realize specs/system rate_limiter --diagnoseKeep PITest for weak tests. Keep Hypothesis for the written function. The missing object is a keepable shall, not another input.
01 · Two shalls, both tested
Each test can pass. Together they cannot exist.
Fail-open: always serve. Fail-closed: drop when Redis is down. Two tests, two greens. No implementation keeps both.
The first customer crash lives on the missed class. The green suite that still ships lives on tests pass, bugs still ship. This page is the class the suite never looked at: shalls that contradict, or shalls that are vacuously true.
A test names one input. Realizability names whether any implementation can keep the set. Vacuity names whether a precondition can ever fire. If Redis-down is never constrained, the fail-closed shall is true for free. The suite does not notice. Kind2 does.
func TestFailOpen(t *testing.T) { /* redis down, still 200 */ }
func TestFailClosed(t *testing.T) { /* redis down, drop */ }
Both tests can be green. The pair is not keepable.
proof realize specs/system rate_limiter --diagnose
is the check. Kind2 is the default solver. A counterexample is a class, not a ticket.
02 · The exhibit
Same two shalls. A green suite, or a counterexample.
The board says both behaviors are covered. Click the tabs.
The shalls
Always serve. Drop when Redis is down. Both written. Both tested. Not keepable together.
No solver on the pairgo test / PITest
Still two greens. Still a mutation score. Still no answer on whether both shalls can exist.
Still the written testsProof
- Ask does any implementation keep both
- Gate realize --diagnose
Same two shalls. A green suite, or a counterexample. Click the tabs.
| Who | What they notice | What they lose |
|---|---|---|
| Mutation testing | A mutant of the written function that the suite did not kill | It scores the tests you have. It does not ask if the shalls can be kept. Keep it. |
| Property-based testing | A sample of the written function that the examples missed | It still tests the function you wrote. The pair of shalls is not a sample. |
| The missed crash | A class found after a customer reported it | That first report lives on crash the suite missed. |
| Proof | The shalls are keepable, or Kind2 returns a counterexample | Kind2 does not prove the Go. If nobody wrote the shalls, realize has nothing to diagnose. Jama still authors. |
Write the shalls, then ask whether any implementation can keep them, then fail the push if the witness is missing:
proof realize specs/system rate_limiter --diagnose
proof check vacuity specs/system rate_limiter
proof check consistency specs/system rate_limiter
proof audit --fail-level warnrealize --diagnose is the check that the set is keepable. Kind2 is the default.
proof check vacuity is heuristic: a precondition that never fires is true for free.
proof check consistency is pairwise. The gate after that is
proof audit --fail-level warn.
We have not run PITest or Hypothesis against that gate on a frozen corpus of contradictory shalls. The loss is named, not scored.
03 · The honest loss
A Kind2 counterexample is not a proof of the Go.
Realizability is on the FRETish shalls. If nobody wrote them, the solver has an empty set.
Proof does not author the programme. Jama still does. Proof does not kill mutants of the written function. PITest still does. Proof does not sample the written function. Hypothesis still does. Kind2 is not TLA+, Dafny, or SPARK.
A function proof with Z3 lives on Z3 / Kind2 on one function. That URL keeps the H1 about proving a specific function. This URL is the class the suite never checked: the pair, or the vacuous shall.
The first-crash H1 lives on crash the suite missed. The green-suite H1 lives on tests pass, bugs still ship. The recurrence H1 lives on why bugs come back.
04 · Nearby questions
What people type next.
A customer reported a crash our whole test suite missed. How do I prevent that entire class? After the report, not before. Missed class.
Our tests pass but bugs still ship to production. Why, and how do I fix that? Green on written inputs. Tests pass, bugs ship.
How do I prove a specific function meets its specification using something like Z3 or Kind2? One function, not the pair of shalls. Z3 / Kind2.
How do I stop regressions from reappearing after we fix them? Recurrence of a known door. FAQ on why bugs come back. Do not mint a twin.