Proof · Engineering practice
What is the difference between test coverage and requirements coverage?
Structural test coverage measures how much of the measured code was exercised. Requirements coverage measures how much of the declared requirement set has the specified evidence. Both need an explicit denominator, and neither percentage guarantees the requirements or assertions are complete.
The import loop runs, but the import promise is untested
Illustrative requirement: import 100,000 valid records without losing any record. A unit test with three rows might exercise every line of the parser. It does not establish behavior at the promised volume or across the complete import transaction.
| Measure | Denominator | What a full result supports |
|---|---|---|
| Statement coverage | Measured executable statements | Every measured statement ran under the selected tests. |
| Branch or condition coverage | Measured decisions and conditions | The specified structural alternatives were exercised. |
| Requirement-link coverage | Declared requirements needing a test link | Each has a recorded test reference. |
| Observed requirement evidence | Declared verification obligations for a specified revision | Required observations exist and pass within their stated scope. |
Do not put “linked” and “observed passing” into one numerator. A skipped test, a stale run and a passing current test are different states. The import acceptance run must check the resulting count and content, not merely a successful process exit.
Review the expected result before counting the test
A test asserting that 99,999 records is acceptable can pass while violating the 100,000-record requirement. Inspect the oracle, inputs and assertions against approved intent. Agent-written tests can be useful; human-written tests can be wrong. Provenance helps review, but it is not the correctness criterion.
Add interruption, malformed input, duplicates or ordering scenarios when the product promises behavior for them. Keep new questions distinct from established obligations until the applicable authority resolves them. A larger suite cannot compensate for an unknown expected result.
Use separate checks for structure, witnesses and execution
go test -cover ./...
proof audit --check acceptance_criteria_witnessed
proof audit --check tests_pass
go test -cover reports Go statement coverage for the selected packages. acceptance_criteria_witnessed checks acceptance-witness structure for testable stakeholder criteria; a child’s unit-test annotation is not an acceptance witness for the assembled system. tests_pass runs the configured project test commands. Review its logs and scope to confirm the import scenario actually ran.
Ordinary tracked witness deferrals remain visible warnings, not passing evidence. Check the effective audit policy when deciding which findings fail a CI job. Provider enforcement and bypass permissions determine whether a failed job prevents merge.
Report gaps alongside the percentage
For one revision, list the requirement inventory, evidence method, linked check, observed status and remaining gap. Count not-run, failed and stale obligations explicitly. Review the inventory against contracts, existing requirements and prior incidents so an omitted promise cannot disappear from the denominator.
Proof connects those obligations to code, hazards and retained change history. The graph helps a later task recover an earlier promise; the old execution still needs a freshness check.
See how to check every declared requirement and MC/DC on Go decisions for the corresponding evidence questions.