Topic · Negative testing

What is negative testing, and how do I keep the reject path true in CI?

Gist

Negative testing checks that invalid input is rejected. Proof fails proof audit --check negative_path_witness_required unless each security-classed requirement has a :negative triple-form witness, not only a happy-path test.

proof audit --check negative_path_witness_required

Keep go test if you already write the cases. Keep SmartBear or Tricentis if you already teach the glossary. Keep Jama if you author shalls. Proof will not count a green happy path as the reject path. Jama still authors.

01 · The happy path is not the reject path

A green authorized request is not a test that the disabled control rejects.

The handler returns 200 when JWT is on. The suite still reads green. The branch that runs when JWT is off returns Skipped. That is a pass-through, not a reject.

Wikipedia, SmartBear, Tricentis, and GeeksforGeeks define the same act: feed invalid or unexpected input and check the software refuses it. That sits next to positive testing on the glossary. Proof does not replace that act. It asks a narrower question: if you wrote a security-classed requirement, is there a test that exercises the precondition-absent shape, or only the authorized happy path?

The motivating miss is mechanical. A request handler begins with if !cfg.JWT.Enabled { return Skipped }. The spec encodes only the happy path. The integration tests exercise the happy path with the control enabled. Review is performed. The audit reports clean. An attacker who flips the config flag bypasses the control entirely. No test ever asserted the unmet precondition was a reject.

The command is deterministic. It does not call a model. It does not generate the negative cases. It refuses a silent pass when an active security-classed requirement has no :negative triple-form witness.

proof audit --check negative_path_witness_required
proof catalog list --domain security

The first command is the reject-path floor. The second lists the live security-domain classes. Green happy-path coverage after a shall that names auth_required is how the skip branch stayed untested.

02 · The exhibit

Same handler. A green authorized suite, or this witness.

JWT-on still returns 200. SYS-REQ-100 never gained a reject-path test. Click the tabs.

CI / happy path

  • JWT on TestEnforce_Authorized_Allows green
  • Credit SYS-REQ-100 marked covered via :nominal
  • Disabled return Skipped, never asserted
Suite closed

This reject path

No test sets JWT.Enabled = false and asserts 401. The skip branch was never a reject.

No witness

CI / happy path

Still the authorized case. Still a green log. Keep the positive test.

Keep the happy path

Proof

  • Ask does SYS-REQ-100 have a :negative witness
  • Out error until SYS-REQ-100:auth_required:negative exists
A green allow is not a reject

Same handler. A green authorized suite, or this witness. Click the tabs.

What the record lists What Proof actually does What a green cell is not
Positive tests Keep them. A shall still wants :nominal on the authorized path Not the reject path. Allowed is not denied.
Witness A test annotated <REQ-ID>:<security-class>:negative that asserts a reject, not Skipped Not a bare // Verifies: on a happy-path test.
Scope Security-classed requirements only. Nominal, error_handling, idempotency stay silent Not ISTQB coverage of every invalid input.
Union One quality :negative on any of the requirement's security classes satisfies this gate Not per-class depth. That floor is obligation_evidence_complete.
CI Default-on error. The only opt-out is a per-spec audit_ignore with a ≥16-character reason Not a signed V&V certificate. Jama still authors.

The witness looks like this on a test file:

// SYS-REQ-100:auth_required:negative
func TestEnforce_JWTConfigDisabled_Rejects(t *testing.T) {
    // arrange: cfg with JWT.Enabled = false
    // act: invoke the request handler
    // assert: 401 / 403 / denied, not Skipped
}

A requirement carrying auth_required and csrf_protection is satisfied on this gate as soon as either class has a :negative witness. Per-class evidence depth remains obligation_evidence_complete. This gate is the security-grade reject-path floor, as a union, so one quality test is enough here.

If the reject path is structurally impossible to drive from the harness, opt out on the spec with audit_ignore and a reason of at least 16 characters. Waived requirements stay visible in the summary. They never look like a silent skip.

We have not run SmartBear, Tricentis, and Proof on the same frozen corpus, and we have not scored this witness as a proof of the Go. The loss is named, not scored.

03 · The honest loss

A green witness check is not a negative-test generator.

The command refuses a silent pass. Proof will not pretend that YAML is a 401, or that a security-class floor is every invalid input.

go test still owns the execution. Wikipedia and SmartBear still own the glossary. Jama still authors the shall. Proof stores the requirement in the repo and fails a silent pass when that shall has a security class and no reject-path witness. It does not invent the invalid inputs. It does not execute the test unless you wrote one. Non-security classes stay silent. The union semantic means one :negative on any listed security class clears this gate, not every class.

Proof is not Tricentis. It is not SmartBear. It is not ISTQB training. Jama still authors. VectorCAST or LDRA still win at a qualified C toolchain.

Random invalid input still lives on fuzz testing. The class of bugs the suite never checks still lives on bugs tests never check. Two implementations on the same input still live on differential testing.

04 · Nearby questions

What people type next.

What is the difference between positive testing and negative testing? Positive tests pin valid input. Negative tests pin invalid input. Wikipedia and SmartBear still win that glossary. proof audit --check negative_path_witness_required is the reject-path floor after the tests exist.

Is Proof an alternative to Tricentis or SmartBear? No. Keep the runner and the glossary. Proof checks that a security-classed shall sits on a real reject-path test. Jama still authors.

Does a happy-path test count? No. A bare // Verifies: on an authorized case witnesses the happy path. The security class needs its own :negative triple.

Does this cover every invalid input? No. The gate is security-classed requirements only. Fuzzing still lives on fuzz testing. Completeness of the input domain lives on equivalence partitioning.

How do I find the class of bugs my tests never check for? That H1 lives on bugs tests never check. This page is the reject-path witness, not that cluster.