Negative testing: verify rejection without forbidden effects

01 / Define the rejection

What should a negative test establish?

A negative case expects rejection or specified failure handling; its test passes when those requirements hold. A failing test is an execution result, not an input classification.

Define the rejection at the layer responsible for it. A parser may reject malformed data before a service sees it; a service may reject a well-formed request because the resource is absent. Calling an internal helper directly can examine that helper, but does not establish that an external caller can reach the same path.

A transaction can provide atomicity for its included database changes. That property does not specify the right response, account ownership or authorization policy. SQLite's atomic-commit documentation concerns a database transaction and its storage assumptions; the selected rejection tests here do not establish universal atomicity. SQLite atomic commit.

Write allowed effects as carefully as forbidden ones. A contract may require a denial audit record while prohibiting a business update. If messages or notifications are involved, include their separate observation boundary: a database rollback cannot establish what another system received. AWS transactional outbox guidance.

02 / Start with the record

Free rejection and effects worksheet

Copy the complete record into a document, issue or repository. It works without Proof, signup or JavaScript. Repeat case and observation fields as needed; mark unknown or unperformed work explicitly. Copying the record does not execute a test.

Copy REJECTION AND EFFECTS RECORD
Record ID / revision / preparer:
Evidence status: NOT RUN
Acceptance status: NOT ASSESSED

A. REQUIREMENT AND AUTHORITY
Requirement ID / revision / authoritative source:
Exact rejection or failure-handling rule:
Requirement owner / person authorized to amend it:
Rejection-owning component or layer:
Bounded claim and decision this record must support:
Executor / reviewer / acceptance authority:

B. ENTRY POINT, ACTOR AND DOMAIN
Supported entry point / adapter / actual implementation call path:
Evidence that this request reaches the intended rejection owner:
Actor identity, permissions and relevant context, or explicit exclusion:
Values the caller controls; values controlled only by the test:
Input class: invalid / absent resource / business rejection / fault
  / authorization denial / valid retry or other successful control
Valid domain, types, units and relevant boundaries:
Exact request or ordered sequence; input artifact identity:
Initial durable state and preconditions; reset/isolation arrangement:
Dependencies, access and controlled external-effect destinations:
Test-only fault seam, injection point and evidence it was reached:
Public reachability claimed? Basis, or explicitly not claimed:

C. INDEPENDENT ORACLE
Exact required response or exception and error details:
Allowed durable effects and independent requirement basis for each:
Forbidden durable effects and independent requirement basis for each:
Required before-state and exact after-state, including identities/rows:
Observer/query/assertion code and revision; why independent of implementation:
Observation milestone, completion/deadline rule and timeout classification:
Transaction/resource state required at return:
External intent, dispatch and delivery observations, if applicable:
Unresolved expected behavior and owner:

D. IMPLEMENTATION AND REPRODUCTION
Baseline / corrected candidate identities, revisions or source hashes:
Harness, cases, schema and oracle identities:
Runtime / dependency versions / platform where relevant:
Connection mode, transaction policy and verified settings:
Prerequisites and environment differences:
Working directory / exact command / case selection:
Output destination; preservation of previous evidence:

E. CONTROLS AND SENSITIVITY
Successful control, expected response and exact effects:
Boundary success control, if relevant:
Known-bad implementation or forbidden-effect challenge:
Expected failure of the strengthened assertion:
Reject-all or other trivial-pass challenge and expected result:
Which controls were actually executed; retained evidence:

F. OBSERVATIONS (repeat per case and gate)
Run identity / source basis / captured runtime:
Case ID / actual request / actual starting state:
Observed response / immediate state / independently observed durable state:
Transaction state at return / evidence of the fault boundary:
Expected versus actual differences and assertion result:
Distinct scenarios / invocations / passed / failed / skipped or blocked:
Raw output, errors, trace and summary locations; hashes where retained:
Failure disposition / implemented correction / replay evidence:

G. DECISION AND RENEWAL
Supported claim and precise acceptance scope:
Failed, missing, unperformed or stale work and its effect on the decision:
Acceptance decision / authority / evidence basis / conditions:
Next action / accountable person or queue / completion or review trigger:
Reassess when requirement, input domain, source, schema, observer, runtime,
transaction policy, dependency, actor context or effect boundary changes:
Initial decision: no acceptance follows from this blank record.

The filled record uses this structure. Resolve expected outcomes before adapting its assertions: generating inputs does not determine correctness. Barr et al., oracle survey.

03 / Inspect a failure and its correction

A rejected transfer that still commits a debit

The authored contract is negative-transfer-rejection-v1. Each invocation starts in S0: [["alice",10000],["bob",2500]], no transfer rows and no active transaction. There are no concurrent callers, prior transfer IDs or triggers.

The runner imports and calls transfer_baseline, transfer_corrected or transfer_reject_all from the same transfer.py. The call uses the case's request fields and separately supplies the internal fault flag. It executes SQLite statements on a fresh on-disk database; no network API or Proof runtime participates.

Declare the cases before inspecting the implementation

These are the seven literal records in cases.json, not a Cartesian product of their values. Every request uses source alice. Every required rejection has status: rejected; both successes require {"status":"accepted","transfer_id":1}.

The implementation's amount check admits exact Python int values, excluding bool, from 1 through 1000000. The executed amounts are only -1, 0, 2000, 10000 and 10001. Unknown sources, other types and the upper limit were not exercised. The supported fixture entry point is the local function; there is no authentication or authorization mechanism in this experiment.

Why the baseline fails

The connection pins LEGACY_TRANSACTION_CONTROL where available and uses isolation_level=None. In that mode, Python does not implicitly open transactions. The baseline issues no BEGIN, so its debit commits before it checks destination existence. Returning unknown_destination cannot undo that completed write. Python transaction control and baseline source.

The second failing case has a valid request but enables an internal test-only seam. Immediately after the real debit, it attempts a zero-amount transfer insert. The real CHECK (typeof(amount_cents) = 'integer' AND amount_cents > 0) raises sqlite3.IntegrityError. SQLite's statement-level ABORT backs out the failing statement; it does not reverse the earlier committed debit. The seam is not a caller-controlled request field or evidence of attacker reachability. CHECK constraints · ABORT semantics.

Both baseline failures return their exact expected response and persist Alice 8000, Bob 2500, with no transfer row. The expected rejected state is Alice 10000, Bob 2500, with no transfer row. Closing and reopening the database exposes the difference. Read both failed observations.

Inspect the implementation or reproduce the retained contrast with the evidence kit.

The implemented correction and replay

transfer_corrected explicitly issues BEGIN IMMEDIATE, validates destination existence before debit, and keeps the debit, credit and transfer record in that transaction. It commits successful writes and explicitly rolls back rejections reached inside the transaction and caught SQLite errors. Invalid amounts return before a transaction opens.

The corrected fault trace records the mechanism, not just the final response:

BEGIN IMMEDIATE occurs outside the protected write body. Its acquisition errors can propagate, and lock contention was not exercised or accepted by this uncontended fixture. SQLite documents that this acquisition can fail with SQLITE_BUSY. SQLite transactions.

The strengthened gate compares exact response data and the test-owned observe_durable_state SQL reads with literal expectations from cases.json. It reads account identifiers and balances and all transfer rows through a new connection, and checks that the operation left no transaction open. It does not use the implementation's arithmetic or diagnostic snapshots as its oracle. A conserved balance sum alone cannot establish which account owns the value or whether the transfer record is correct.

The response gate validates setup and response while only recording post-operation state. The reject-all stub fails both successful controls, showing that refusing every request cannot satisfy this contract. These controls establish sensitivity to the two observed state defects and blanket rejection. Four-gate summary · Reject-all observations.

Reproduce the contrast

Extract the ZIP into a new directory. Use Python 3.10+ with its standard-library sqlite3 module and a writable temporary directory. No package installation, credentials, network service or Proof signup is needed. From the extracted kit root:

Copy python3 reproduce.py

The default rerun-output/ must be absent or empty. To retain an earlier run, use the documented alternative:

Copy python3 reproduce.py --output-dir another-run

Wrapper exit 0 means the expected passing and failing children reproduced; it does not accept the baseline. Keep new observations separate from observed/; the README describes their files. Python 3.10+ is a prerequisite, not a tested version matrix.

Filled rejection and effects record

This record summarizes the actual frozen source and primary observations. Technical evidence is filled; real-project authority and adoption work remain unassigned.

Copy REJECTION AND EFFECTS RECORD
Record basis: negative-testing-sqlite-rejection-v1
Evidence: observed fictional teaching fixture, retained capture
Project acceptance: not assessed; no customer or production decision

A. REQUIREMENT AND AUTHORITY
Requirement: negative-transfer-rejection-v1, contract.json
Basis: authored teaching rule, with literal cases.json expectations.
Rule: the five declared rejections return their exact response, preserve
both account rows and all transfer rows, and leave no transaction open.
The two successes return accepted/transfer_id 1 and exact required effects.
Rejection owner: selected local transfer function in transfer.py.
Requirement/acceptance authority: no actual project owner assigned.
Executor: retained reproduce.py -> run_cases.py campaign.
Real-project reviewer: unassigned; no approval is inferred.

B. ENTRY POINT, ACTOR AND DOMAIN
Call path: reproduce.py starts run_cases.py; IMPLEMENTATIONS selects the
imported transfer_baseline, transfer_corrected or transfer_reject_all.
run_case calls implementation(db, **case["request"],
  fail_after_debit=case["test_fault_after_debit"], events=events).
Actor: local harness; source account alice is not an authenticated actor.
Authorization and public/network reachability: not assessed.
Request fields: source_id, destination_id, amount_cents.
Internal test control: fail_after_debit; never an external request field.
Implementation amount domain: exact Python int, excluding bool, 1..1000000.
Executed domain: precisely the seven cases below; no Cartesian expansion.
Each call starts in its own fresh on-disk database at S0:
  accounts = [["alice",10000],["bob",2500]]; transfers = []
  active transaction = false; no prior IDs, triggers or competing callers.
No external-effect destination exists in this fixture.
Fault boundary: real zero-amount transfer INSERT after real debit;
CHECK constraint raises IntegrityError before credit/normal record write.

C. INDEPENDENT ORACLE
All amounts and balances are integer cents.
Allowed rejection-side durable effects: none in accounts or transfers,
as required by contract.json. Diagnostic trace events are not DB writes.
Forbidden effects: any account-row or transfer-row difference from S0.
Rejection expected response: {"status":"rejected","reason":REASON}
  missing-destination: alice -> ghost, 2000; unknown_destination
  insufficient-balance: alice -> bob, 10001; insufficient_balance
  zero-amount: alice -> bob, 0; invalid_amount
  negative-amount: alice -> bob, -1; invalid_amount
  record-failure-after-debit: alice -> bob, 2000, internal seam;
    processing_failed
Every rejection's exact expected after-state is S0.
Successful controls, each independently from S0, fault disabled:
  successful-transfer: alice -> bob, 2000
    response = {"status":"accepted","transfer_id":1}
    accounts = [["alice",8000],["bob",4500]]
    transfers = [[1,"alice","bob",2000]]
  successful-full-balance-transfer: alice -> bob, 10000
    response = {"status":"accepted","transfer_id":1}
    accounts = [["alice",0],["bob",12500]]
    transfers = [[1,"alice","bob",10000]]
Independent basis: literal cases.json values from the authored contract.
Observer: run_cases.py::observe_durable_state, using its own SQL:
  SELECT account_id, balance_cents FROM accounts ORDER BY account_id
  SELECT transfer_id, source_id, destination_id, amount_cents
    FROM transfers ORDER BY transfer_id
Milestone: capture response/immediate state/transaction flag; close the
operation connection; reopen the same DB and read durable rows.
Gate: setup matches; response matches; reconnected rows match;
in_transaction_after is false. No implementation calculation is reused.
Timeout/delivery policy: not exercised; no external calls or delivery claim.

D. IMPLEMENTATION AND REPRODUCTION
Baseline and corrected functions coexist in frozen transfer.py.
transfer.py SHA-256:
df8d4cf13056eb283edfce4d8791d77fe5d0cf6bc8f0ac53b05fe6f41ed1580d
Harness/schema/case/contract identities: manifest.json and SHA256SUMS under
/assets/negative-testing/research-rerun/.
Primary runtime: Python 3.14.7; SQLite 3.53.4.
Connection: LEGACY_TRANSACTION_CONTROL explicitly pinned when available;
isolation_level=None. Captured autocommit_attribute=-1, foreign_keys=1,
journal_mode=delete, synchronous=2, in_transaction_before=false.
Policy: baseline has no BEGIN; corrected uses BEGIN IMMEDIATE and explicit
COMMIT/ROLLBACK. Invalid amounts return before BEGIN. BEGIN acquisition
failure is outside the accepted boundary.
Prerequisites: Python 3.10+, standard-library sqlite3, writable temporary
storage, extracted kit; no network, package install or Proof dependency.
Portable command, cwd extracted kit root: python3 reproduce.py
Output: rerun-output/ must be absent or empty; preserve observed/ unchanged.
Alternative: python3 reproduce.py --output-dir another-run
Actual primary child commands, cwd recorded as . containing run_cases.py:
/opt/homebrew/opt/python@3.14/bin/python3.14 -u run_cases.py --implementation baseline --gate response
/opt/homebrew/opt/python@3.14/bin/python3.14 -u run_cases.py --implementation baseline --gate effects
/opt/homebrew/opt/python@3.14/bin/python3.14 -u run_cases.py --implementation corrected --gate effects
/opt/homebrew/opt/python@3.14/bin/python3.14 -u run_cases.py --implementation reject-all --gate effects --cases positive

E. CONTROLS AND SENSITIVITY
Both successful controls above ran against baseline and correction.
Known-bad baseline: response gate passes, effects gate detects the two
specified persisted-state mismatches.
Reject-all challenge: both successful controls fail response and state.
Fault trace confirms the real constraint error; corrected trace confirms
debit inside a transaction and an explicit rollback before return.
This checks sensitivity to these selected failures, not all possible faults.

F. OBSERVATIONS
Primary capture: 2026-10-28T10:06:31.421386+00:00 through
2026-10-28T10:06:31.645946+00:00.
Cases missing-destination and record-failure-after-debit:
  baseline response matches; after reconnect accounts are
  [["alice",8000],["bob",2500]], transfers=[]; transaction closed.
  corrected response matches; after reconnect S0; transaction closed.
Corrected fault trace: temporary 8000/2500, transaction true;
  IntegrityError; explicit rollback to 10000/2500, transaction false.
baseline-response: 7 passed, 0 failed, child exit 0.
baseline-effects: 5 passed, 2 failed, child exit 1.
corrected-effects: 7 passed, 0 failed, child exit 0.
reject-all-positive: 0 passed, 2 failed, child exit 1.
Distinct scenarios: 7, comprising 5 rejections and 2 successes.
Fresh-database invocations: 23 = 7+7+7+2. No skipped selected cases.
Wrapper result: expected contrast reproduced; not baseline acceptance.
Raw evidence base: /assets/negative-testing/research-rerun/observed/
  baseline-response.stdout.json; baseline-effects.stdout.json;
  corrected-effects.stdout.json; reject-all-positive.stdout.json;
  matching .stderr.txt files; reproduction-summary.json.
Summary SHA-256:
9dcfe32769035500260fd90406a0d1e8eaf97ff0141aa62e02e0670501259607
Temporary databases were deleted after their reconnected state was captured.

G. DECISION AND RENEWAL
Supported: corrected implementation matches these seven literal cases
from S0 under the recorded uncontended connection policy.
Baseline: rejected by effects gate. Correction: observed, not merely proposed.
Unperformed: concurrency, lock contention, crash/power loss, disk full,
corrupt database, authorization, notification/delivery, restart/idempotency,
Proof/model/solver execution, production behavior and universal proof.
Actual project acceptance and responsible people: unassigned.
Proposed next actions before adopting or extending this example:
  Engineering lead: assign implementation executor and evidence reviewer.
  Requirement owner: approve real rejection, retry and allowed-effect rules.
  Developer: connect the real supported entry point and rerun its assertions.
  Storage/security/integration owners: plan applicable excluded boundaries.
  Acceptance authority: decide only after required current evidence exists.
Renew evidence after changes to rule, domain, implementation, schema,
cases, observer, runtime, transaction policy or proposed deployment context.

04 / Work in a useful order

How to design and assess a negative test

1. Agree the rule and owning boundary. Identify who decides acceptable behavior, which component rejects, and whether the case is invalid input, absent data, a state-dependent refusal or a fault. Resolve competing error reasons and valid retry behavior before writing assertions.
2. Choose inputs and a successful neighbor. Select a concrete representative of each relevant rejection class and the nearby valid behavior it must preserve. Record actor context and caller control. Keep internal fault injection separate from ordinary request fields.
3. Prepare identifiable state and configuration. Capture the initial rows or objects, pin transaction and dependency settings, and isolate independent cases. For a sequence requirement, retain the whole sequence instead of resetting away the history that matters.
4. Write the oracle independently. Specify exact response semantics and every allowed or forbidden effect. Name the observer, fields and completion milestone. Derive expected values from the requirement, not by calling the implementation's helper.
5. Execute the supported path and retain observations. Record the revision, command, actual input, response and post-operation state. Verify that the intended layer or injected boundary ran. Setup failure or an unrelated exception is not a successful negative test.
6. Challenge, repair and replay. Make the check reject a named forbidden effect and a trivial implementation that refuses valid operations. Correct the responsible code, replay the failure and run successful controls and affected cases. Preserve the failing record beside the correction.
7. Make the scoped decision. Report distinct cases separately from repeated invocations. Assign missing work and an acceptance authority. Reassess evidence when its requirement, implementation, observer or environment changes.

05 / Check before relying on the result

Negative-testing checklist

Ready to run

• The rejection rule, owner, input class and supported entry point are identified.
• Actor permissions and public versus test-only controls are explicit.
• Initial state, environment and transaction settings are reproducible.
• Response and effect expectations have an independent requirement basis.
• Successful controls, a sensitivity challenge and an evidence destination exist.

Ready for a decision

• The intended implementation and rejection or fault boundary were reached.
• Observations include the required identities, rows and completion milestone.
• The asserted response and durable effects both match the approved rule.
• Selected bad controls fail for the intended reason; valid controls succeed.
• Failed, unperformed and stale work has owners; acceptance has a recorded scope and authority.

06 / Avoid misleading evidence

Mistakes that hide a rejected operation's effects

07 / Keep intent and observed evidence connected

Use Proof without confusing a witness with an assertion

Inspect the annotation, configured result and connection to implementation code separately. Then review whether the actual assertions protect the specified rejection response and allowed/forbidden effects; these checks do not supply that review or an acceptance decision. The implementation basis was inspected on 28 October 2026.

Technical policy details at this revision. A child can credit its parent only if active, explicitly linked by Satisfies, and carrying an overlapping security obligation. Inapplicable suppressed classes are removed; suppressing all excludes the requirement. A check-level audit_ignore waiver is counted separately. A valid deferral with a reason and tracking reference records outstanding work instead of a missing-annotation error. Ordinary deferred work warns; work classified as acceptedKnownIssue is listed separately and need not warn. Invalid deferrals do not excuse absence. None creates a test or passing result.

For result checking, findings normally warn; they become errors for a test file within the source or target paths of a migration/mirror record marked complete. This “complete mirror” does not mean complete test coverage. Certain not-run tests become findings when other credited tests in the same file have observed results; missing suite declarations and test files also matter. Other not-run cases can remain counts.

For code connection, no implementation target makes the check inapplicable. An annotation naming a catalog evidence type marked coverage_trackable=false is exempt; the pinned help names integration, fuzz and differential evidence. This does not verify whether the test ran in another process. Unsupported language, unresolved scope or missing build/run/profile information may be reported as an advisory limitation rather than a confirmed finding; a passing headline can include those gaps.

Run tests in your repository or CI and retain outputs against the candidate and requirement. The SQLite fixture ran no Proof checks.

For a written eligibility predicate, plan cases in Spec Lab, review the interpretation and add the state/effect obligations here. Its hosted analysis produces planned checks; it does not run the repository or SQLite harness.

08 / Choose the missing observation

Which method fits the rejection rule?

Use positive controls to challenge trivial satisfaction. This practical check is not formal non-vacuity analysis, which the IBM abstract discusses for temporal models.

10 / Apply the record to another decision

Two examples beyond the transfer fixture

1. A repeated request that should succeed

Illustrative requirement and proposed checks; unperformed here. Suppose the contract says that the same request key and payload must return the previously recorded success without another business effect. The repeat is a valid positive case, even if it arrives after a client timeout.

Prepare the completed operation and its recorded response. Submit the same key and payload through the supported interface. Compare the returned outcome with that record, and assert no additional debit, transfer row or event intent. A matching response alone leaves duplicate effects unresolved.

The same key with a different payload is a separate policy decision. Have the owner specify its outcome before classifying the case. Also plan expiry, restart and simultaneous retries if those belong to the product's contract. None of those mechanisms exists in the SQLite teaching fixture.

2. Authorization denial with a required audit event

Illustrative requirement and proposed checks; unperformed here. Suppose a tenant may update only its own document, and a denied update must preserve the document while appending a denial audit record. “Every table stays unchanged” would reject correct behavior under that chosen contract.

Prepare an authenticated actor from another tenant, an existing protected document and the supported update request. Require the specified denial; compare the document's identity, owner, contents and version with the before-state; separately check the permitted audit record's actor, target and reason. Run an owner's successful update as a control.

Establish that the actor can supply the tested document identifier through the real entry point. A direct helper call with a fabricated internal identity does not demonstrate that public path. The transfer fixture supplies no authorization evidence for this example.

12 / Continue with the missing evidence

Related guides

• Equivalence partitioning: separate input classes that require different outcomes.
• Fuzz testing: explore malformed inputs with a failure-sensitive oracle.
• Property-based testing: express behavior across a declared generated-input domain.
• Software verification plan: assign the remaining methods, owners, criteria and acceptance work.