Correctness labels

01 · One task

One claim, with the evidence on it.

SYS-REQ-110 says what Set must do past the end of an array. This record is the task cut from that claim, and the command that checks it.

DEFECT-260727-WWWY

episode ep-DEFECT-260727-WWWY · task_family bug_fix · reward toolchain

Task: Fix Set() so a beyond-length array index appends instead of destroying the array. Before the fix, Set({"a":[1,2,3]}, 99, "a", "[9]") returned {"a":[99]}. After, it returned {"a":[1,2,3,99]}.
Subject: github.com/buger/jsonparser @ 6454f95de679fdd0c623a6183f7cff82da56b442 (MIT). The zip is the post-fix tree. The affected revision is public and outside it. findings/INDEX.md says how to reach it.
Intent: SYS-REQ-110 is approved. reviewed_by: human:buger, 2026-07-26. When Set targets an array index [N] with N >= len(array), the parser shall append at the end and return the mutated document. It shall not overwrite existing elements or panic. It rests on STK-REQ-005. The claim is formalized in FRETish.
Failure class: boundary · high Silent data loss. Sibling hazards on the same requirement: nested_mutation (medium) and element_type_partition (high).
Available context: The requirement neighbourhood (SYS-REQ-110, SYS-REQ-009, STK-REQ-005), the implementation (parser.go:Set), the historical failure (DEFECT-260727-WWWY, and KI-4 for the top-level partition), and the public tests named in the record.
Verifier: A deterministic go test pin. No LLM judge. The exit code is the verdict.
go test -count=1 -timeout 120s -run 'TestSetBeyondLengthScalarArrayPreservesElements_SYS110|TestMCDC_SYS_REQ_110' .
Run it through re-run/verify.sh --pin DEFECT-260727-WWWY.
Success: The known witness passes (TestSetBeyondLengthScalarArrayPreservesElements_SYS110). The MC/DC row for this requirement passes. The sibling partition (KI-4, top-level arrays) keeps its own status.
Provenance: PROVENANCE.json records the MIT licence, the subject commit, the producer (Proof / ReqProof), the review policy, and a SHA-256 for the content root and for each of the 201 payload files.

Real record · Read from the public sample: hierarchy/SYS-REQ-110, findings/problem-reports/DEFECT-260727-WWWY.yaml, and serialization/findings.jsonl. Section 05 opens the requirement node.

02 · Why this exists

Bug pairs show the fix. Labels keep the claim.

A bug pair only shows broken, then fixed. A correctness label keeps the claim, the ways that claim can fail, and the check that proves it. You can rerun that check after delivery.

What you have today

Source trees, scraped or licensed
Bug tags and preference pairs
Unit tests, or an LLM judge as the reward
Intent latent in commits and tickets
Failure modes that surface after incidents
Hard to re-verify after delivery

Models see symptoms, not contracts.

What a correctness label adds

The requirement hierarchy above the code
A formula on each machine-checkable claim, checked before any code is graded
Hazards on each claim: worst case and severity
Links a machine can read, from requirement to code and tests
Condition-level coverage, and formal checks where that path is mature
Findings with runnable reproducers, on the same graph
A person validates every finding. The reviewer is named in the file.

The requirement is the source of truth.

03 · Scope

What this is, and what it is not.

Evidence on claims in code you license. Not a pile of unlabeled files.

What it is

A versioned requirement graph over one component you license. You keep the files: the claims, the hazards, the code and test links, the findings with reproducers, and a script that reruns the verdicts. Each task has a deterministic verifier and the provenance to rerun it.

Bulk code

Not bulk unlabeled code. You already license the trees. Proof does not compete with scrapers or corpus brokers.

Agent output

Not unverified agent output. Where a machine check is claimed, the item passed one. Findings carry runnable reproducers.

Client work

Not another buyer’s engagement. Client and private audit work is never licensed as data. Eligible subjects are the trees you already license, and open-source corpora whose annotations we authored and can license. The jsonparser sample is the second kind.

Our own history

Not Proof’s own source history. We label your licensed component. We do not resell our repository as training data.

04 · The unit

The unit is a claim with its evidence.

A task is cut from a requirement. Parent and child links are explicit. Walk from any claim to its code, its tests, and its hazards.

Stakeholder intent
STK-REQ. Why the product exists, at acceptance level.
System requirement
SYS-REQ. What must always hold.
Component contracts
Software (SW-REQ) and interface (INT-REQ), when the graph needs them.
Hazards
How this claim fails. Worst case and severity on every shipped requirement.
Code and test annotations
Implements and verifies links. A machine can read them.
Formula and evidence
Machine-checkable claims carry a FRETish formula. FRETish is structured English from NASA’s FRET. Proof compiles it to temporal logic and checks realizability, consistency, and vacuity before code is graded. Coverage, findings, and validation follow.

Two kinds of requirement, marked in the file. A machine-checkable requirement carries a FRETish formula and is graded by the checks on it. A prose-only requirement states intent that no formula expresses. It carries formalization_strategy: informal, an empty fretish field, and it is discharged through the machine-checkable requirements under it. In the jsonparser sample, all 116 system requirements are machine-checkable and all 7 stakeholder requirements are prose-only. Every package prints that split. A requirement count is not an executable-check count.

SYS-REQ-110
One requirement node.
Satisfies
STK-REQ-005 · the parent intent.
Hazards
Worst case and severity, per failure class.
Implemented by
parser.go - Set().
Verified by
set_spec_test.go · mcdc_supplement_test.go.
Formula
FRETish, checked before code is graded.

Fig. 01 · One node from the sample, SYS-REQ-110, and the edges stored beside it. The next section opens the file.

The full model → How the evidence gets made →

05 · The claim

The claim, and the evidence on it.

The data-loss class in jsonparser’s Set helper. The package ships these files. The links open them on GitHub.

SYS-REQ-110system · approvedcomponent: parser

Set beyond array length shall append. It shall not silently overwrite.

It satisfies the mutation-helper intent under STK-REQ-005. It sits beside SYS-REQ-009. It is the leaf for the data-loss class.

Set() in parser.go carries the requirement IDs that govern it. The tests name the requirement they verify. Three graded hazards hang off this claim.

boundary · high
Silent overwrite. An index past the array’s length destroyed elements the caller never addressed, and returned a mutated document with no error. MC/DC and negative boundary tests discharge it.
nested · medium
Wrong offset. A beyond-length index inside a nested container writes at the wrong offset. A sibling can be overwritten, or the JSON can come out malformed.
partition · high
Whole-array loss. Scalar-first arrays took a replace-container branch, and the entire array was destroyed. ✓ fixed The regression is pinned under the same claim as DEFECT-260727-WWWY.

The affected code had 100% MC/DC at the time. Every condition in the checked decision logic was exercised. Exercised is not correct. This class still produced two published misses. The accounting is on public proof →

specs/system/requirements/SYS-REQ-009.req.yaml

id: SYS-REQ-009
status: approved
priority: shall
component: parser
fretish: the parser shall always satisfy !set_path_is_provided | set_target_exists | set_creates_missing_path | set_returns_updated_document | set_returns_not_found_error
formalization_strategy: fretish
traces:
  satisfies:
    - STK-REQ-005
  verified_by_extra:
    - mcdc_supplement_test.go
    - set_spec_test.go
  reviewed_by: human:buger
obligation_hazards:
  - class: boundary
    worst_case: 'Set on an array-index path component [N] where N >= len(array) silently overwrites element 0 or another existing element the caller did not address, destroying data and returning a mutated document with no error (PR #286 regression class).'
    severity: high

Verbatim fields from the public node. reviewed_by names the person who validated it. That name is data in the file. The full node on GitHub ↗

The root-cause write-up ↗ The full specification tree ↗

06 · Hazards

Failure is graded, not just pass or fail.

Most corpora record a failure only after an incident. Here every shipped requirement records how it can fail, before anything breaks.

Class

A reusable failure type, such as boundary, partition, or fail-open control. The same catalog recurs, so negatives compare across the corpus.

Worst case

The concrete consequence on this requirement. “Silently overwrites an element the caller did not address, destroying data with no error” is a training signal. “Data loss” is a slogan.

Severity

Low, medium, high, or critical. It is authored triage, not a CVSS score, and the file says so.

For a model, this sits between the code and the incident. It is graded, per requirement, and machine-readable. In an evaluation set, it is the failure category on the task. The data-loss record in section 05 is one entry.

07 · The samples

Download the samples. Rerun the verdicts.

Two downloads from the public jsonparser audit. The label sample is the fixed tree: eight real defects, not a seeded demo. The environment package adds the base revision, a reset, an isolated toolchain, and a hidden grader.

The label-and-evidence sample

serialization/ 123 requirements, 12 findings, 8 demo episodes (one per defect), 670 traces and hazards, plus SCHEMA.md
PROVENANCE.json subject commit, licence, producer, and a SHA-256 for the content root and each file
re-run/verify.sh deterministic go test pins. The command is below.
corpus/ 123 requirement files, 8 defect records, 4 known-issue records, MIT sources, reproducer tests, and the root-cause write-up
hierarchy/ the SYS-REQ-110 spotlight from sections 01 and 05. README.md is the entry point.

proof-jsonparser-sample-v1
version 1.0.1 · 442 KB zip

Download the public sample

sha256 04b646a3ea14fb3938e14db8153d850e2acdaed7ecc309415b7bb3a30ca96729
checksum sidecar · Go 1.21+ is all verify.sh needs

what “rerun the verdicts” means here

$ unzip proof-jsonparser-sample-v1.zip && cd proof-jsonparser-sample-v1
$ ./re-run/verify.sh                             # all eight pins
$ ./re-run/verify.sh --pin DEFECT-260727-WWWY    # one finding

ok    github.com/buger/jsonparser  0.33s
==> OK - pin(s) green on packaged (post-fix) sources

Go 1.21+ is enough. verify.sh runs go test. No network. No LLM judge. The exit code is the verdict. Green means those checks pass on the packaged revision. It does not mean the library is correct. Unwritten behaviour stays outside the checks. The zip is the post-fix tree, so green means the fixed tree still holds the pin. A red-before run needs the pre-fix commit, public and outside the zip.

The environment package

tasks/<task_id>/ 8 bug-fix tasks in 3 fix families, Go only. Each has the instruction, metadata, public checks, hidden go test pins, and one reference patch per family. Difficulty is null. There is no authored difficulty label.
repo/ One base tree per family, without .git, without the Proof defect records, and without the base tests that asserted the pre-fix behaviour. Removed tests: 13 in f0958b5, 3 in 3005d5b, 0 in 753cda0. Declared per family and per task.
environment/ setup.sh, reset.sh, and policy.json. Network is none. Allowed commands are go build, go test, go vet, and gofmt. The image is golang:1.22.
verifier/ grade.sh scores 1.0 or 0.0. No LLM judge.
quality-report.json 8 candidates, 8 accepted, 0 rejected. Hidden checks fail on the base and pass after the fix, 8 of 8. Three repeated runs agree. Reset is byte-identical in 16 of 16 trials. Docker with network none matches the host.
intent/ and PROVENANCE.json 123 requirements, dependencies, hazards, history, the contamination declaration, and a SHA-256 for the content root and each file.

proof-jsonparser-env-v2
version 2.0.0 · 676 KB zip · 60 files

Download the environment package

sha256 15fd42b87945f8d3cf28c3c015439a2d805c50165c3c3d6c0b341e87c5d2403d
checksum sidecar · bash, Go, python3, tar and zstd on the host, or the grader image from its Dockerfile

what “run one task” means here

$ unzip proof-jsonparser-env-v2.zip && cd proof-jsonparser-env-v2
$ WORK=$(mktemp -d)
$ environment/setup.sh task-260727-wwwy "$WORK"   # the base tree, extracted to $WORK/repo
$ verifier/grade.sh task-260727-wwwy "$WORK"      # reward 0.0 on the base tree: the hidden check fails
# ... the agent edits files under $WORK/repo ...
$ verifier/grade.sh task-260727-wwwy "$WORK"      # reward 1.0 when the hidden check and the public suite pass
$ environment/reset.sh task-260727-wwwy "$WORK"   # back to the base tree, byte-identical

This is the Set() defect from sections 01 and 05, on its base revision. The grader never modifies $WORK/repo. It copies the tree, restores every base *_test.go, copies the hidden checks in fresh, runs the hidden filter, then the public suite without the hidden checks. Reward is 1.0 only when both pass. No network. No LLM judge. Red before and green after are measured for all 8 tasks in quality-report.json. The patch is tasks/task-260727-wwwy/reference/fix.patch.

What the downloads are, and are not. The label sample shows the schema, provenance, evidence graph, and post-fix checks. The environment package adds the base revision, a reset, an isolated toolchain, and a hidden grader. Neither has a private subject or a held-out split. jsonparser and its fixes are public. A clean evaluation set needs a private subject.

123 requirements · 7 stakeholder + 116 system 116 of 123 carry a FRETish formula · count them in specs/ ↗ 8 defect records on master · 2 credited to outside reporters 2 misses published

The same corpus, elsewhere: the specs on GitHub ↗ · the register in the portal ↗ Open findings on branch proof-demo are demonstrations. The real register is buger/jsonparser master. Read the labels first. · The full accounting on public proof.

08 · Deliverables

One graph. Three products.

The Software Intent Graph is where the tasks come from. The three cuts follow a data team’s order: evaluate, curate, then post-train.

01 · evaluation sets

Evaluation sets

Held-out tasks grounded in requirements, with failure categories and deterministic graders.

02 · training data

Training data

Requirements, code links, failure modes, defect histories, and expert-reviewed outcomes. The graph is the source for supervised or preference pipelines. It ships as a correctness label package: versioned files you keep and can re-check after delivery. No SFT or preference format ships in the public sample.

03 · RL environments

RL environments

Resettable codebase tasks. An agent can inspect the repository, change it, run tools, and receive a versioned reward. The public package ships the base revision, a reset, an isolated toolchain, and a hidden grader for 8 tasks on a public subject. Customer environments are scoped after a label pilot.

Why the reward signal is different. Agents game graders. They read future commits, special-case inputs, and hunt a judge’s blind spots. A deterministic, versioned verdict takes out model-judge subjectivity and makes common exploits inspectable. It does not stop reward hacking. A high reward means the declared checks held in the stated environment. It does not mean the program is correct everywhere. You can rerun the check after delivery.

Formal checks run through Kind2 / Z3 where that path is live. Condition-level coverage and reproducers carry the rest. What “verified” means here →

FRETish · Kind2 · Z3 · hazard analysis · MC/DC

09 · Start

Start with one component.

Your code stays yours. We license the labels, not your source.

Your code

You supply a codebase you already licensed. Provenance of that input stays on your side. We do not resell your source. No other buyer’s engagement enters your package.

Our labels

We license the derived layer only: annotations, hazards, and evidence.

Deterministic verdicts

The verdicts come from tools, not from a model: toolchain exit codes, coverage, and formal checks. If your policy restricts which models may draft, drafting runs on the stack you allow.

The engineering mirror of this custody story →

One component, boxed.

One component of a codebase you license, or a clearly licensable open-source slice. We recover the history, write the requirements, link the code and the tests, record coverage and hazards, and attach findings with reproducers. A machine checks every shipped item. A person validates every finding.

Duration: 2-3 weeks of calendar time. A tight box, not a monorepo firehose.
Commercial: Fixed fee, quoted in the first reply. Complexity or exclusivity changes the quote before work starts.
Success: Measured on the pilot’s own list. No counts are promised up front.

Accepted tasks per component

Verifier determinism

Environment reset reliability

Task rejection rate

Expert-review minutes per accepted task

Exploit and reward-hacking findings

Cost per accepted task or environment

In production, cost per accepted requirement node against market rates is the secondary metric. A node is accepted when it carries its hazards, its code and test links, and the evidence its kind requires.
Out of scope: Entire monorepo dumps. Third-party finding dumps sold as samples. Client or private audit work. The sale of Proof’s own product history. Environments are the third product, and they are scoped after a label pilot.

The same engine installs Proof for product teams in roughly four weeks, on the audit page. The pilot above is the data-team shape. It is shorter, and it is judged on label quality.

Email is enough to start. hello@reqproof.com.

Proof was founded by Leonid Bugaev. It operates the Proof platform and its named assurance practice. Twenty-plus years in engineering. Head of Engineering at Tyk API Management. Judge the work by the public jsonparser corpus above.

A correctness label is evidence on a claim.