Proof · Engineering practice
How to verify AI-generated code before merge
Verify AI-generated code against agreed behavior, including the promises the change must preserve. Review the test’s expected answer, exercise relevant failure cases on the candidate revision, and retain results and gaps. A second agent’s agreement does not establish that the software works.
Six checks before accepting an agent-written change
- Establish the intended behavior. Name the requirement or decision that supplies the expected answer. Resolve conflicts between the prompt and existing promises.
- Follow the impact. Check affected callers, dependencies and neighboring requirements. Include behavior the patch must preserve.
- Inspect the assertions. A test can repeat the same mistaken assumption as the code. Derive expected outcomes from the agreed behavior.
- Run relevant checks at the candidate revision. Include boundaries, failure paths and regressions. Confirm that the intended tests actually ran.
- Record results and gaps. Keep the command, environment, revisions and output. Separate observed failures from missing or stale evidence.
- Apply the acceptance policy. Keep exceptions and their authority explicit. A passing test does not grant merge permission or authorize weaker requirements.
Use the PR worksheet to retain these answers. The depth of each check depends on the consequences of the change.
An Enter key that answered the wrong question
In our Omarchy work, a newly summoned menu prompt could leave an old uninstall confirmation active. Enter intended for the prompt could uninstall the selected app. The fix dismisses the old confirmation when the new request arrives.
| Review question | Recorded evidence |
|---|---|
| Does the regression reproduce? | The regression test fails against 821ae58 and passes against fix 09bfcda. |
| What ran? | Ten assertions exercising actual QML functions under Node with stubbed dependencies. A separate recording demonstrates the behavior in the real shell. |
| What remains separate? | The broader shell suite had the same 17 failing files before and after on the test host. A passing focused test does not make that full suite green. |
The retained requirement was in review in the October 8, 2026 record. Requirement approval and maintainer acceptance are separate from these observations.
A discount test can agree with the wrong cap
Illustrative example: the approved requirement caps a discount at 40%. An agent implements a 50% cap and writes a test expecting 50%. That passing test confirms the implementation’s chosen value; it contradicts the approved requirement.
| Input | Approved result | Copied assertion |
|---|---|---|
| 30% discount requested | 30% applied | 30% applied |
| 45% requested | 40% applied | 45% applied |
| 60% requested | 40% applied | 50% applied |
An agent can also write the correct tests, using the approved 40% cap as its independent expected result. Sharing a session with the implementation is a reason to inspect correlated assumptions, not a reason to discard every generated test. The same circular mistake can occur in human-written code.
Recover the agreement before checking the patch
Read existing requirements, customer promises, documentation, earlier changes and relevant incidents. Code provides evidence of current behavior, which may be deliberate or accidental. Treat recovered intent as a candidate until its authority is established, and reconcile it with retained obligations.
An authorized person or policy resolves a conflict such as 40% versus 50%. An implementation agent must not silently change the cap, remove its regression test or relax the policy just to obtain a passing result. See approval policy and the current initialization default.
Use each check for the question it can answer
Static analysis can detect relevant implementation hazards. Property tests can express the cap across generated inputs. Mutation testing can reveal assertions that fail to notice a changed cap. Structural coverage can show which branches ran. None of these techniques becomes invalid because an agent prepared it; each needs an appropriate oracle and scope.
proof audit --check tests_pass
proof audit --check acceptance_criteria_witnessed
The first command runs the project’s configured test commands. The second checks the recorded acceptance-witness structure for testable stakeholder criteria. An acceptance annotation is not an execution result, and a linked test can still have the wrong assertion. Inspect both the test and its current run.
For the discount example, exercise values below, at and above 40%, then verify any relevant rounding, authorization and refund behavior. Keep failed and not-run scenarios visible instead of collapsing them into a single green score.
Ask for a result you can inspect
Put this record in the pull request. Link to the actual run so another engineer can examine the inputs and assertions.
Download the plain-text template, or select and copy it here. This worksheet works with your existing tools.
Change and purpose:
Intended behavior and authoritative source:
Existing behavior to preserve:
Candidate code and requirement revisions:
Check, inputs and expected outcome:
Observed outcome and output link:
Relevant behavior not checked:
Open decisions and their owner:
Acceptance decision and applicable policy:Accept the evidence under the existing policy
Retain code and requirement revisions, command, environment, output and the authorized decision. If using CI as a merge gate, configure the job as a required check on the candidate revision, restrict bypasses and protect the policy and regression evidence from unauthorized weakening.
The public jsonparser postmortem shows why coverage can miss an unmodeled case. Use that limitation to challenge the requirement and evidence plan, rather than treating a passing suite as global correctness.
Bring the change you are unsure about
Bring the proposed behavior, the patch and the tests you already have. Proof can audit the change alongside your existing workflow, or help take it through implementation and verification as a factory engagement.