The suite
- Asked the inputs already in the file
- This push zero failures
Topic · Green suite
Our tests pass but bugs still ship to production. Why, and how do I fix that?
Gist
The suite asked the inputs it already knew. Proof fails the merge when an approved shall has no witness on this branch. Sentry still files the instance. PITest still mutates the suite.
proof audit --fail-level warnA door that re-opens after a fix lives on why bugs come back. A customer crash that names a missing class lives on crash-suite. This page is the green build.
01 · The green lie
A passing suite is a sample. Production is the rest of the class.
The tests you wrote asked whether those inputs survived. They did not ask whether the promise you sold still has a witness.
The recurrence question lives on why the same bugs come back. That page is a door that re-opens after a fix. The incident question lives on a crash the suite missed. This page is earlier: CI is green, the ticket is not from a customer yet, and the hole is already in the tree.
Two different jobs sit in the same sentence. proof audit --check tests_pass still owns running the configured suite. If that check is red, the evidence chain is already broken. The production-ship problem is the other job: the suite returned zero failures, and the approved shall was never in the denominator.
proof audit --check tests_pass
proof realize specs/system ./... --diagnose
proof audit --fail-level warn
Kind2 realizability answers a different question from a unit test. A test asks whether this input survived. proof realize asks whether there exists any implementation that can keep the approved shalls. If a guarantee is false for some input the component cannot refuse, the solver returns a counterexample. That counterexample is a class, not a ticket.
02 · The exhibit
Same helper. Two questions CI can ask.
The suite can stay green while the shall is unwitnessed. Click the tabs.
The shall
The clamp at 0 was never a case. Production hit it.
Not in the sampleThe suite
Still green. Still the same three cases.
Still greenProof
- Ask realize --diagnose on the set
- Gate audit --fail-level warn
Same helper. Two questions. Click the tabs.
| Who | What they notice | What they lose |
|---|---|---|
| Sentry / the ticket | The instance after it shipped | The class was already open in CI. Sentry still files the ticket. |
| PITest / mutation | A test that does not kill a mutant | The mutant is still inside the suite you wrote. The missing shall is not. |
| go test -cover | A line ran | The denominator is lines, not the approved shall. That split lives on coverage vs requirements. |
| Proof | The shall still has a witness, or Kind2 returns a class | 100% MC/DC on the written decision still misses a partition nobody named. |
We have not run Proof against Sentry, PITest, and a coverage gate on a frozen corpus. The loss is named, not scored. Keep the suite. Keep mutation. Keep the ticket tracker. None of them is the merge gate for an approved shall.
03 · The honest loss
A green Proof audit is not a closed production incident.
Kind2 does not invent the shall. 100% MC/DC on the written decision still misses a partition that was never a condition.
If nobody signed what the helper is for, there is nothing for proof audit --fail-level warn to fail on except the suite you already have. Jama still wins at programme authoring. Sentry still files the instance. PITest still mutates tests. proof audit --check tests_pass still has to run the suite; a red suite is a different failure.
The coverage denominator stays on test coverage vs requirements coverage. Recurrence stays on why bugs come back. The first customer crash stays on crash the suite missed. The MC/DC instrument stays on MC/DC for Go.
04 · Nearby questions
What people type next.
Why do the same bugs keep coming back in our codebase? Recurrence after a fix. Why bugs come back.
A customer reported a crash our whole test suite missed. How do I prevent that entire class? The first report. Crash the suite missed.
What's the difference between test coverage and requirements coverage? Two denominators. Coverage vs requirements.
We have high test coverage but still get escaped defects. What are we missing? Same denominator cluster. Do not mint a twin. Stay on the coverage page.