Verifier-backed SWE tasks from code you license

01 · Input → output

What goes in. What ships.

You supply one component you are allowed to use. Proof builds the intent graph around it. What ships is that graph as versioned files, with the check for each task.

You supply

A codebase you own or license
One component of it, named and bounded
Its history: commits, tickets, incidents
Its tests and build steps

Proof builds

Approved requirements and their dependencies
Failure modes and hazards, graded per requirement
Links an agent can read, from requirements to code and tests
Verified defect records with reproducers

What ships

A versioned dataset with deterministic checks
Tasks with graders: one command, exit code is the verdict
Environment material: what the task can see and run
Provenance: licence, subject commit, per-file hashes

Fig. 01 · What you supply, what Proof builds, and what ships. The public samples come from the jsonparser audit.

02 · The task unit

One codebase task, end to end.

One task from the public jsonparser packages. Set destroyed an array when the index ran past the end. The check fails on the base revision and passes after the fix.

DEFECT-260727-WWWY

fixed · holds on the packaged tree

Kindbug fix · hard · demo split · toolchain reward, not a model score

Task

Fix Set() so that a beyond-length array index appends instead of destroying the array.

Reproducer from the record: Set({"a":[1,2,3]}, 99, "a", "[9]") returned {"a":[99]} before the fix and {"a":[1,2,3,99]} after.

Subject

github.com/buger/jsonparser @ 6454f95de679fdd0c623a6183f7cff82da56b442 · MIT

The label sample is this post-fix tree. The broken revision is the base revision in the environment package.

Intent

SYS-REQ-110 · approved · reviewed by buger on 2026-07-26

When Set targets an array index [N] with N >= len(array), the parser shall append at the end and return the mutated document, not overwrite existing elements or panic.

It formalizes STK-REQ-005. This defect broke that promise.

Failure class

boundary · silent data loss · severity high

The hazard on SYS-REQ-110. The record calls the surface data integrity. Two other hazards sit on the same requirement: nested mutation at medium, and element type at high.

Available context

The agent sees SYS-REQ-110, SYS-REQ-009, STK-REQ-005, the Set function in parser.go, this defect, and the public tests named in the record. KI-4, top-level arrays, is a separate case. The hidden check is not in that view.

Base revision

ae6fe746a98e5978263c5adcd139ada2171da1b5 · family f0958b5 · ships in the environment package as repo/base-f0958b5.tar.zst · the fix is f0958b571b35

No .git, and no Proof defect records. The 13 tests that asserted the bug are removed. families.json and metadata.json declare each removal. TestSetBeyondLengthScalarArrayPreservesElements_SYS110 fails on this tree and passes after reference/fix.patch. Three runs agree. Reward is 1.0 or 0.0. No model judge.

Verifier

go test -count=1 -timeout 120s -run 'TestSetBeyondLengthScalarArrayPreservesElements_SYS110|TestMCDC_SYS_REQ_110' .

The exit code is the verdict. No model judge.

Success

The known witness passes: TestSetBeyondLengthScalarArrayPreservesElements_SYS110. The MC/DC row for this requirement passes. KI-4, top-level arrays, keeps its own status.

This holds on the packaged tree. A failing run needs the pre-fix commit.

Provenance

PROVENANCE.json names the MIT licence, the subject commit, the producer, and the review policy. It stores a SHA-256 for the content root and for each of the 201 payload files.

The label sample is the post-fix tree. A pass there means that tree still holds the pin. findings/INDEX.md says how to reach the broken revision by hand. The environment package also ships the base revision. The hidden check fails there and passes after the reference patch. Both results are in quality-report.json.

Real record · Read from proof-jsonparser-sample-v1, version 1.0.1. The base revision is read from proof-jsonparser-env-v2, version 2.0.0. The defect record ↗ · the requirement ↗

The same task, on the full page → The postmortem behind this defect →

03 · Package schema

The package, as files.

Each package is one versioned tree. A manifest hashes every file. The first tree is the environment package, 2.0.0. The second is the label sample, 1.0.1.

Package schema as shipped in 2.0.0 and 1.0.1

proof-jsonparser-env-v2/ · environment package · 2.0.0 · 60 files

README.md
PROVENANCE.json       licence, subject commit, producer, per-file SHA-256
LICENSES.json
checksums.txt
quality-report.json   8 accepted, 0 rejected, reset 16 of 16
intent/               123 requirements, dependencies, hazards, history
environment/          Dockerfile golang:1.22, setup.sh, reset.sh, network none
verifier/             grade.sh, reward 1.0 or 0.0, no model judge
repo/                 families.json, base-753cda0.tar.zst, base-f0958b5.tar.zst, base-3005d5b.tar.zst, no .git
tasks/<task_id>/      8 tasks: TASK.md, metadata.json, public_checks.json, hidden_checks/, reference/fix.patch

A task names its family, base commit, and fix in metadata.json. The family names the base tarball, the patch, and the removed tests in repo/families.json. requirement_ids point into intent/requirements.json. Grader output follows verifier/result.schema.json.

proof-jsonparser-sample-v1/ · label sample · 1.0.1 · 201 payload files

README.md
PROVENANCE.json
INVENTORY.json
LICENSE               MIT, the subject licence
NOTICE.md
serialization/        requirements, findings, traces, episodes
re-run/               verify.sh
findings/             8 problem reports, 4 known issues
hazards/
hierarchy/            STK-REQ-005, SYS-REQ-009, SYS-REQ-110
mcdc/
rl/                   ENVIRONMENT.md
corpus/               123 requirement files, source, and tests

serialization/SCHEMA.md maps the joins. A finding points at the requirement through affected_requirements, and at the pin through reward_pin.command. A requirement points at its tests and its code.

04 · Quality controls

What a package declares.

Each line is a control the environment package declares, or a limit it leaves open.

Whether the repository was ever public. declared · stays open yes: the subject is public, so this is a demonstration, not a held-out set
Whether the task appeared in commits or issue text. declared · stays open yes: the task text and the fixes are on the subject’s public master
Whether future commits are reachable. declared no: the agent tree ships without .git
Whether outbound network is disabled. declared yes: policy sets network to none. The Docker grade was measured that way.
Whether hidden tests are visible to the agent. declared no: they live outside the agent tree, and the grader copies them in fresh on every grade
Whether the agent can modify the verifier. declared no: verifier/ is outside the agent view, and the grader restores every base test file before it runs
How reference patches are isolated. declared under tasks/<task_id>/reference/, outside the agent view, one patch per family
How task families are split. declared by fix commit: 3 families for 8 tasks
Whether base tests that assert the bug were removed. declared yes, listed per family and per task: 13 in f0958b5, 3 in 3005d5b, 0 in 753cda0
Which checks detect special-casing or reward hacking. open no search was made; the report says so

Nine of the ten are declared, in metadata.json and in PROVENANCE.json. The public subject is the limit. A clean evaluation set needs a private one. No search for reward hacking was made. The label sample only says that the repository is public, that verify.sh needs no network, and that no model judge sits in the verdict. Every pin is named in findings/INDEX.md.

What the quality report carries

Tasks generated / accepted / rejected: 8 / 8 / 0
Hidden checks compile on base, fail on base, pass after the fix: 8 of 8
Public suite passes on base and after the fix: 8 of 8
Reproducer determinism: three repeated hidden runs agree, 8 of 8
Environment reset: byte-identical tree in 16 of 16 trials
Verifier agreement: Docker (golang:1.22, network none) matches the host, 1 task
Requirements: 123 · 28 approved · 95 in review

What it does not carry

Human-review minutes: null, not measured
Known grader exploits: not searched

This is quality-report.json in proof-jsonparser-env-v2. Counts and booleans only. No percentages. A pilot returns the same report. The two empty fields are measured then, and listed under Pilot.

05 · The public downloads

Two public packages, one subject.

Both downloads are free. Both come from the public audit of jsonparser, a zero-allocation Go JSON library long in production. The label sample is the graph and the post-fix checks. The environment package holds eight bug-fix tasks, with base revisions, a reset, and a hidden grader.

The label-and-evidence sample

123 requirements: 7 stakeholder and 116 system. Those 116 carry a FRETish formula.
12 findings. 7 primary fixed defects and 1 related, in 8 problem-report files, plus 4 known issues.
8 bug-fix episodes. 7 in the demo split, 1 in the eval split. The reward is a toolchain result.
670 trace rows and 201 payload files. MC/DC is 100% of decisions and conditions.
re-run/verify.sh needs Go 1.21+ only. No network. No model judge.
rl/ENVIRONMENT.md says what the demo episodes are, and what they lack.

proof-jsonparser-sample-v1
version 1.0.1 · created 2026-08-05 · 442 KB zip

Download the public sample

sha256 04b646a3ea14fb3938e14db8153d850e2acdaed7ecc309415b7bb3a30ca96729
checksum sidecar · the full technical page →

The environment package

8 bug-fix tasks. 8 candidates, 8 accepted, 0 rejected. Go only. Difficulty is null. No authored label.
3 families, split by fix commit. One base tarball and one reference patch each.
Base trees have no .git and no Proof defect records. Tests that asserted the bug are removed: 13 in f0958b5, 3 in 3005d5b, and 0 in 753cda0.
Hidden checks fail on the base and pass after the fix, 8 of 8. Three repeated runs agree.
Reset is byte-identical, 16 of 16. Docker (golang:1.22, network none) matches the host on 1 task.
Reward is 1.0 or 0.0. No model judge.

proof-jsonparser-env-v2
version 2.0.0 · created 2026-09-04 · 676 KB zip · 60 files

Download the environment package

sha256 15fd42b87945f8d3cf28c3c015439a2d805c50165c3c3d6c0b341e87c5d2403d
checksum sidecar · the same package on the full page →

The label sample shows the schema, the provenance, the evidence graph, and the post-fix checks. The environment package adds the base revision, a reset, an isolated toolchain, and a hidden grader. Neither has a private subject or a held-out split. jsonparser and the fixes are public. A clean evaluation set needs a private one.

Rerun the verdicts Go 1.21+ · no network · no LLM judge

Rerun the pins from the zip

$ unzip proof-jsonparser-sample-v1.zip && cd proof-jsonparser-sample-v1
$ ./re-run/verify.sh                             # all eight pins
$ ./re-run/verify.sh --pin DEFECT-260727-WWWY    # the task above
$ ./re-run/verify.sh --list                      # the pin ids

Go 1.21+ and nothing else. The script runs go test on the packaged sources. No network. No model judge. Exit 0 passes, exit 1 is a test failure, and exit 2 is bad arguments. A pass means the fixed tree still holds the pin. The pre-fix commit is public and outside this zip. The environment package ships it.

Run one task setup → grade → reset · Go · no network · no LLM judge

One task, from the package README

$ unzip proof-jsonparser-env-v2.zip && cd proof-jsonparser-env-v2
$ PKG=$(pwd); WORK=$(mktemp -d)
$ environment/setup.sh task-260727-wwwy "$WORK"       # the base tree, extracted to $WORK/repo
$ cat tasks/task-260727-wwwy/TASK.md                   # the instruction the agent receives
# ... the agent edits files under $WORK/repo ...
$ verifier/grade.sh task-260727-wwwy "$WORK" --out "$WORK/result.json"
$ cat "$WORK/result.json"                              # {"reward": 1.0 | 0.0, "hidden": {...}, "public": {...}}
$ environment/reset.sh task-260727-wwwy "$WORK"       # back to the base tree, byte-identical

The grader never edits $WORK/repo. It copies the tree, restores every base *_test.go, and copies the hidden checks in fresh. The public suite runs in a copy without those checks. Reward is 1.0 only when both pass. Otherwise it is 0.0. No network. No model judge. The result follows verifier/result.schema.json.

The base fails. The reference patch passes.

$ verifier/grade.sh task-260727-wwwy "$WORK" --out "$WORK/base.json"   # reward 0.0: the hidden check fails on base
$ (cd "$WORK/repo" && git apply --whitespace=nowarn "$PKG/tasks/task-260727-wwwy/reference/fix.patch")
$ verifier/grade.sh task-260727-wwwy "$WORK" --out "$WORK/fix.json"    # reward 1.0: hidden check and public suite pass
$ environment/reset.sh task-260727-wwwy "$WORK"

On the host: bash, Go, python3, tar, and zstd. Use the zstd CLI, or compression.zstd in Python 3.14. The reference patch is git diff output. git apply works outside a repository. With Docker, build environment/Dockerfile from the package root and run it with --network none. That image is the grader. It holds the hidden checks and the reference patches, and it is never the agent view.

06 · Pilot

Pilot on your codebase.

One component, a fixed fee, and a report you can check.

Input: One component of a codebase you are authorized to use: a buyer-licensed codebase, or a clearly licensable open-source slice.
Duration: 2-3 weeks of calendar time.
Commercial: Fixed fee, quoted in the first reply. Complexity or exclusivity changes the quote before work starts.
Pilot success: Accepted tasks per component

Verifier determinism

Environment reset reliability

Task rejection rate

Expert-review minutes per accepted task

Exploit / reward-hacking findings

Cost per accepted task or environment
Out of scope: Monorepo dumps. Third-party finding dumps as samples. Client or private audit work. The sale of Proof’s own product history.

Email is enough to start. Write to hello@reqproof.com.

The first reply quotes the fixed fee. It asks which component, what licence you hold, and what the task data is for.

The long version is correctness labels, trust, and about.

Verifier-backed SWE tasks from code you license.