Topic · Flip fixtures exist

Flip fixtures exist

Gist

A FLIP scenario you opted into and never generated is missing evidence. Proof runs proof audit --check flip_fixtures_exist. A coverage report is not that hop. Jama still authors.

proof audit --check flip_fixtures_exist

Keep the unit tests if they still catch the samples. Keep Jama if it already holds the shall. Neither one asks whether tests/<component>/tc-*.json exists for a component you listed as fixture-backed.

01 · The silent empty pass

Verify can print clean because scenarios were never in scope.

You can ship a FRETish guarantee, measure MC/DC on the Go, and still have no generated FLIP files. This hop stays quiet until you opt a component in.

The check is flip_fixtures_exist. It is VERIFY-stage. Default severity is warning. It is disabled by default. Enable it with project.checks.flip_fixtures_exist.enabled: true, and list the component in project.verification.fixture_evidence.components. Then the hop asks whether FLIP scenario files exist. It does not measure independence. The instrument stays on MC/DC coverage for Go. The Z3-example hop stays on property fixtures exist.

Empty fixture_evidence.components is a pass: no components declare fixture-backed evidence -- FLIP fixtures not applicable. Opted-in components with no FLIP-eligible requirement (no FRETish, or not a guarantee) is also a pass: no declared fixture-evidence components have FLIP-eligible requirements -- FLIP fixtures not applicable. Those passes are silence, not a scenario set. Retired, superseded, and rejected requirements are not in the active set. A requirements load error, or a tests/ path that is not a directory, is fail: the hop cannot decide whether fixtures apply. Zero tc-*.json under an opted-in component with formalized guarantees is warning: N formalized reqs across M components, K components missing FLIP fixtures.

The expensive miss is a review that treats generated scenarios as part of the pack, then CI that never wrote them. The audit then reports that coverage was fine, because the instrument hop closed, or because this hop never ran. This hop is the gate that says the scenarios exist, or names the components that still have none.

This is not whether the Go decision was independently covered. That hop is MC/DC coverage for Go. A 100% MC/DC report can still have zero tc-*.json files. The help file teaches that first.

# proof.yaml
project:
  checks:
    flip_fixtures_exist:
      enabled: true
  verification:
    fixture_evidence:
      components: [firewall]

# tests/firewall/ is empty
# proof audit --check flip_fixtures_exist
# 1 formalized reqs across 1 components, 1 components missing
# FLIP fixtures
# missing: firewall (1 reqs)
# warning: the scenarios were never generated

The fix is to generate the scenarios, or to drop the component from the list if they were never a review artifact. Run proof testgen specs/system <component> --engine heuristic. Then re-run VERIFY. Do not copy a sample JSON by hand to silence the checker. A quiet hop after you ignore the warning is still not a fixture set. The default warning does not block advancement.

proof audit --check flip_fixtures_exist --verbose
proof testgen specs/system firewall --engine heuristic
proof help flip_fixtures_exist

02 · The exhibit

Same FRETish guarantee. Silent empty pass, or this hop.

One last green VERIFY on firewall. The Go report was green. Nobody wrote tc-*.json. Click the tabs.

The stamp

  • Ask did the Go decision hit independence
  • Stamp mcdc_coverage last pass. tests/firewall empty
  • Why the instrument hop closed. nobody asked for scenarios
Instrument hop green

This hop

Nobody asked whether the opted-in scenarios exist. A coverage report is not a tc-*.json file. The finding kind is this hop.

Need unread

The stamp

Keep the live coverage. Keep the Jama cell. That is not this hop.

Keep the record

Proof

  • Ask do opted-in FLIP scenario files exist
  • Out flip_fixtures_exist, 1 formalized req, 0 tc-*.json
Scenarios missing

Same FRETish guarantee. Silent empty pass, or this hop. Click the tabs.

Surface What they do What Proof does What we lose
Green tests The last samples that still unioned. Whether opted-in tc-*.json files exist under tests/. We do not rerun the suite here. A green stamp is not this hop.
MC/DC coverage Whether each condition independently affects the decision. Whether scenario files exist after you opted the component in. Not the instrument. See MC/DC coverage for Go.
Property fixtures exist Whether opted-in Z3 examples exist as z3-*.json. This named hop: optional FLIP scenarios for listed components. Not the Z3-example gate. See property fixtures exist.
LDRA / VectorCAST An avionics toolchain that already owns MC/DC on C. A warning the audit can name next to missing tc-*.json. We do not replace LDRA. We have not run a frozen VectorCAST corpus. See Proof vs LDRA.
Jama cell A shall, and a parent if you type it. A warning the audit can name next to the missing scenarios. Not Jama's V&V. Jama still authors. We have not run a frozen Jama pack.

The teaching graph is still one guarantee whose Go report was green while tests/firewall/ stayed empty. Close it by generating the scenarios, or by removing the component from fixture_evidence.components if they were never a review artifact. Do not add another sample only to silence the checker. With no opted-in component the empty pass still prints, and the scenarios did not go anywhere. They just were never in scope.

# scenarios generated for the opted-in component
# proof testgen specs/system firewall --engine heuristic
# wrote tests/firewall/tc-001.json

# proof audit --check flip_fixtures_exist
# 1 FLIP fixtures across 1 components (1 formalized reqs)
#
# VERIFY is allowed to move on. a pass now sits on files

The instrument hop stays on MC/DC coverage for Go. The Z3-example hop stays on property fixtures exist. The rewrite hop stays on characterization testing and mirrors. Jama still authors. Proof vs Jama.

03 · The honest loss

Proof names the missing scenarios. It does not write them, and it does not prove the Go.

A quiet proof audit --check flip_fixtures_exist can still mean the hop was never enabled. Jama still authors.

Warning when opted-in FLIP-eligible guarantees have no tc-*.json. Fail when requirements cannot be loaded, or when tests/ is not a directory, so the hop cannot decide whether fixtures apply. Empty fixture scope is a pass, not a fail: the hop does not invent scenarios. Disabled by default: if you never enable it, CI never asks. The hop does not write the JSON. It does not add a shall. It does not prove the Go. It does not run Kind2. It does not measure independence. A quiet hop is not a proof that the authored guarantee is the one you meant, only that every opted-in component has scenario files, or that none were in scope. It does not say the fixtures still match the FRETish. That freshness hop is fixture_staleness_clean. The default warning does not block advancement. We have not scored this floor against a frozen Jama pack or a VectorCAST corpus. The loss is named, not scored.

The instrument hop stays on MC/DC coverage for Go. The Z3-example hop stays on property fixtures exist. The rewrite hop stays on characterization testing and mirrors. The engagement stays on software correctness audit. Jama still authors.

04 · Nearby questions

What people type next.

What is flip fixtures exist? Same question. Same URL.

Is this MC/DC coverage? No. That hop is whether each condition independently affects the decision. This hop is whether opted-in scenario files exist. See MC/DC coverage for Go.

Is this property fixtures exist? No. That hop is Z3 examples as z3-*.json. This hop is FLIP scenarios as tc-*.json. See property fixtures exist.

Is this characterization testing? No. That page is rewrite-with-a-ledger. This hop is generated FLIP files for a listed component. See characterization testing and mirrors.

Is this fixture staleness? No. Staleness is whether existing files still match the formalization. This hop is whether the files exist at all.

Does a quiet hop prove the Go matches the shall? No. The hop observes files. It does not prove the Go.

Does empty fixture_evidence fail this hop? No. That is a pass. A pass with nothing in scope is not a fixture set.

Does missing tc-*.json fail the merge? No. The check keeps warning severity unless you raise it in proof.yaml.

Is Proof an LDRA alternative for the instrument? No. LDRA still owns the avionics toolchain. Proof vs LDRA.

Is Proof a Jama alternative for the shall? No. Jama still authors. Proof vs Jama.