AI-written tests
- Source same context window
- Pass means they agree with the function
Topic · Agents
How do I verify code that an AI agent wrote is actually correct?
Proof does. Recover the intent, pin it, and run proof audit. CodeQL, Stryker, and Hypothesis check the code against itself or against generic rules. They never check it against what you meant.
proof auditNot a replacement for static analysis. Keep those tools. They do not hold the shall.
01 · The circular proof
A green build can mean the agent agreed with itself.
When an agent writes a function and then writes the test, both artifacts came out of the same context window. The test passes because it was derived from the code it is checking. Agreement was never the question. The question is whether either one matches what you meant.
Intent
Not in the loop. Nobody approved the cap.
OutsideThe loop
Still green. Agreement was cheap.
Green suiteProof
- Command proof audit
- Held to signed requirements
Same agent. Two checks. Click the tabs.
- Static analysis finds bug classes it has a rule for. It cannot know your discount cap was 40%.
- Mutation testing asks whether the suite would notice a change. If the suite encodes the wrong behavior, the score still looks great.
- Property tests are closest — and if the agent writes the property too, you are back inside the loop.
02 · Outside the loop
Put the spec where the model that wrote the code cannot edit it.
Recover intent first, and have a human approve it. Measure the AI’s tests against that spec, not the implementation. Force condition-level coverage on the decision logic models learn from happy-path examples. Turn each finding into a red reproducer the agent has to clear, and an independent gate agrees — not the model.
On the public jsonparser audit, 100% MC/DC still let a defect ship, because the missing partition was never in the denominator. Coverage of the code is not a check of the intent. That is the whole argument, and it is on Agents as the product shape.