Topics / Engineering practice

Make CI check the promises that changed.

A useful correctness gate connects a revision to its requirements and current evidence. It should explain failures and missing support, not just add another green badge.

Keep evidence production separate from the policy decision.

CI runs builds, tests and other configured checks. A policy decides which outcomes permit progress. Make the revision, configuration and result artifacts explicit so the policy evaluates the intended candidate.

Illustrative example

The test job passed, but the evidence was old.

A pipeline reuses a cached report after a dependency change. The check command exits successfully because the file is readable. That does not establish the new dependency’s behavior. The gate needs the report’s production basis, not only its presence.

Inspect a real local failure before wiring the gate.

In the retained-obligation fixture, reducing retry-key retention to 12 hours breaks a 24-hour promise. The same test command exits 0 at A, 1 at B and 0 after the repair at C. Proof’s selected tests_pass check observes the same sequence.

project:
  commands:
    test: python3 verify.py --config config.json --output .evidence/verification.json --junit .evidence/tests.xml
  checks:
    tests_pass:
      affected:
        mode: full

This excerpt deliberately forces the configured full test command. The complete fixture configuration also supplies its project, specification and JUnit settings.

proof audit --check tests_pass --no-cache --format json

--check runs only that check. --no-cache does not force a nonempty affected-test plan. For committed branch changes, select the intended comparison base when using affected planning; the default base is HEAD. The full-command setting makes this small example’s execution explicit.

The actual failure log reports two failing cases. The package is blocked; the repaired package stays not_assessed because important acceptance inputs are absent. Nothing was posted to a CI provider, and no repository protection rule was exercised.

Audit’s default failure threshold is warn, unless project configuration or a command flag changes it. Check severity, exit policy and provider enforcement are separate settings. A green command alone is not authorization to merge.

Introduce the gate in a controlled order.

Choose one component and its consequential requirements.
Run the relevant checks and retain artifacts on the candidate revision.
Inspect warnings and gaps with the team before enforcing policy.
Make blocking rules explicit and review exceptions.
Test that missing, failed or stale required evidence is visible.
Keep a recovery path for tool failures without disguising them as passes.

Avoid turning every available check into a blocker at once. A noisy policy invites workarounds and obscures material failures.

Use Proof for the connected assessment.

Proof’s audit and verification commands assess configured requirements and evidence. Individual checks differ in scope, defaults and severity. Configure the CI exit policy deliberately; a warning in an audit is not automatically an enforced merge rule.

The graph supports investigation of what changed and what needs renewed evidence. Your repository protection rules and release process remain responsible for enforcing the chosen delivery policy.

Try it on your software

Add one meaningful gate to an existing pipeline.

Bring a recurring regression and the CI evidence you already collect.

Plan a CI pilot ↗ Inspect public evidence ↗