Proof-Gated Agent Patching (Patch the Planet's Regression Audit)

Sources#

Summary#

The vault's security-research pages measure how agents find bugs. This page is about the other half of the same campaign: what happens when agent-assisted fixes for those bugs land upstream. Anish Naik and Marcelo Morales of Trail of Bits (Your AI agent fixed the bug, but what else did it change?, 2026-10-21, case-study) went back over every Patch the Planet patch that one of their engineers authored and a maintainer merged, and counted what the patches broke. Patch the Planet is the consultancy's open-source hardening campaign with OpenAI, whose bug-finding side is described in LLM-Driven Vulnerability Research and Loop Engineering. The engineers wrote the patches through thousands of Codex sessions.

The tally: 21 confirmed regressions from "over 130" merged PRs, so at most about 16% of merged patches broke something. 16 were functional, 1 performance and 4 build/test. None broke a security protection. A regression here means "a new defect introduced by a change that breaks previously correct and supported behavior along a path a real user or attacker can reach."

The post turns that audit into two plugins in the public trailofbits/skills repository, patching-bugs and regression-hunt, and their design is the transferable part. Every step takes and returns a fixed schema. Every claim needs a proof of concept run against a live, built instance of the project. A fix counts as proven only if reverting it brings the bug back.

Evidence note. case-study, tier kept. The regression tally is a first-hand review of the firm's own merged work, which is unusual candour for a vendor post. Everything else is a product launch. The plugins are Trail of Bits' own, the campaign is co-branded with OpenAI, and the agents are Codex. The 25% false-positive figure is called "preliminary" by the authors. They evaluated it themselves, on the same 21 regressions the audit produced and the plugin was designed from, against a baseline prompt of one sentence. No run count, absolute false-positive counts, recall figure or cost is given. The September post that first reported Patch the Planet regressions ("1Password's AI patching benchmark is misleading", 2026-09-15) is linked but not in this vault.

The audit, and why it was noisy#

An agent searched each merged patch for possible regressions, confirmed each one with a PoC run before and after the change, and handed the results to an engineer. In a single run it produced 58 candidates, of which the engineers confirmed 21 (~36% precision). So close to two in three candidates that reached a human were not real regressions, even after the agent's own before/after PoC step. Different runs also take different routes and return different candidates, so each extra run adds more to sort by hand. The authors draw two conclusions: patches need a standard way to be written, and regressions need a standard way to be checked, "with written rules for what counts as a regression."

Read against the review cluster. Security fixes are exactly the PR type CRBugBench found hardest to review: bug-fixing PRs carry 0.521 times the detection odds, which its authors explain, without testing it, as attention going to the fix rather than to what the fix breaks. Patch the Planet's breakage pattern fits that explanation. Every regression broke ordinary behaviour (functionality, performance, the build), and none broke the security property the patch was written for. The audit does not test the explanation either, but it is the first population in the vault made up entirely of security fixes with a regression count attached.

Against the follow-up-fix rate. Takerngsaksiri et al. find that 4.5% of merged agent PRs draw a verified fix within 30 days (3.68% on the shared window), against 2.34% for humans. Here, actively hunting for regressions found one in no more than about six merged patches. The two figures are not comparable. One is engineer-directed security patches in mature projects, found by a PoC-confirmed hunt with no time limit stated. The other is five agents' PRs in ≥500-star repositories, found by waiting for a fix PR. But the gap suggests what the fix-PR construct already concedes on its own page: waiting for someone to fix a regression is a floor on how many regressions there are.

Four design principles#

The authors list four principles, drawn from the merged-patch review and the engineers' Codex sessions:

Opinionated guidance in as few words as possible. Record only the firm's judgment of what a good patch and a real regression are, and leave the agent wide latitude otherwise. This is the same lesson as the campaign's earlier goal-prompt finding, where a one-sentence description of the bug class surfaced bugs that the exact root cause did not.
Modular steps. Both plugins are sets of skills that work alone or together.
Strict input and output schemas. Each step takes and returns "the claim, the code revision, the required check, the raw evidence, and open questions." The stated purpose is to let each step "verify the previous step's work rather than trusting its summary." That is a hand-off contract aimed at the failure in which a summary reports success the work did not achieve (Failures That Look Like Success).
A live proof of concept for every claim. The PoC runs against "a built, running instance of the project, reached through the interfaces a real user would use." If a step cannot produce one, its result is inconclusive. It fails closed rather than defaulting to a verdict, the same rule ExecCritic: Learn to Test, Test to Improve's harness applies to tests.

regression-hunt#

Figure 2's pipeline: an orchestrator spawns agents per PR (Prepare → Hunt), which enumerate candidates by tracing the diff through its callers to real interfaces. Each candidate then goes through its own chain: a Reproduce agent (before vs after, on a live deployment), a Replay agent that reruns the PoC in a fresh context, and an Adjudication agent that decides, before the result goes into a Report.

Two features carry the design:

A definition of what a regression is not. At adjudication the burden of proof is on the agent: it must show the suspected regression is not a false positive, an incomplete fix, a latent bug, a pre-existing bug, "and so on." The plugin reports a regression only if every gate passes. It is a written negative definition, aimed at the 64% of candidates the audit threw away.
The threat model as global context. If the project has a THREAT_MODEL.md, the orchestrator passes it to every step, "an attempt to require each step/agent to refer back to the threat model before reaching its conclusion."

The result, and how far it reaches. On the 21 confirmed regressions, regression-hunt produced 25% fewer false-positive findings than a baseline Codex agent prompted to "review the PR for regressions," with a 95% confidence interval of 14–38%. The stricter evidence bar also cost recall. Some known regressions came out borderline: the plugin found a possible regression but could not gather enough evidence to confirm it. The post gives no count of borderline or missed regressions for either arm, so the precision-recall trade cannot be priced. Three limits apply on top of that. (1) The test set is in-sample: the 21 regressions come from the audit the plugin's rules were written from. (2) The baseline is a one-line prompt with no PoC requirement, so the comparison prices the whole pipeline against almost nothing, not any single gate. (3) The unit behind "25% fewer" (findings per run, per PR) is not stated. The relative reduction is the only published figure, and the absolute false-positive rate after it is unknown.

patching-bugs#

Figure 3's pipeline: an orchestrator calls a Reproduce agent (baseline PoC), a Validate agent (independent rerun), a Patch agent (invariant-guided repair) and a Prove agent (independent checks). The result goes to regression-hunt, which is mandatory, and then to a report and verdict, with a maintainer hand-off "when ready". The post names three emphases beyond reproduction on live deployments and independent adversarial review:

The three-arm proof. The same reproduction runs three times: the original code must fail, the patched code must pass, and a complete revert must fail again. Fail-before / pass-after is the usual fail-to-pass test check. The revert arm adds that the bug comes back when the patch alone is removed. That rules out a pass caused by "an unrelated environment or test change" made alongside the patch. Wiki reading: if the original and reverted code were byte-identical with an unchanged environment, the third arm would repeat the first. It does its own work only when the patch arrives bundled with other changes, which is the normal case for an agent's working tree.
Invariant-guided repair. The patch fixes the violated security invariant, not the single reported instance, so that it also covers variants in the same bug class.
Mandatory regression analysis. Every patch goes through regression-hunt.

What is not evaluated. No figure is given for patching-bugs at all. The authors list an evaluation (does the patch fix the bug and its variants without regressions) as future work. Whether the three-arm proof and the mandatory hunt lower the ~16% regression rate is unknown (see Open Questions).

Project knowledge the plugins read#

The post asks maintainers for two files:

THREAT_MODEL.md — assets whose compromise matters, trust boundaries, access controls, supported configurations, what is and is not a vulnerability, and the evidence a valid bug report needs. regression-hunt passes it to every step.
PATCHING_GUIDELINES.md (or AGENTS.md) — how to build and test, how big a fix should be and where it goes, how to present changes, and what a regression test must show. patching-bugs reads it.

For GPAC, Trail of Bits engineers wrote a threat model and a project-specific patching skill. liboqs and validator.js have also published threat models "in accordance with our guidance." This is a new kind of file on the receiving end of agent contribution volume. The maintainer writes down what counts as a vulnerability and a regression for the agents patching the project, so the rules are written once in the repository rather than argued out in each PR. The next planned option goes the other way: maintainers will be able to loosen regression-hunt's rules to surface more candidates when coverage is worth more false positives to them.

Where it sits#

A deployed assurance plane, built by a patch author. The Replay agent's fresh context and the Validate and Prove agents' "independent" reruns are the deployed form of Independent Assurance Plane's separation of build from evidence, and of the decoupling rule in Optimizer–Evaluator Decoupling. Two differences from that framework matter. Here one vendor's pipeline both writes and proves the patch, with model and organization unchanged throughout, and only the context is fresh. And the adjudicator's "not a pre-existing or latent bug" gates have no counterpart in a seeded-fault study, where every fault is known in advance.
Precision, measured where the review benchmarks measure only recall. CRBugBench scores recall and never penalises a false positive. This post reports only a false-positive reduction and treats recall loosely ("borderline"). Each measures what the other leaves out, and neither gives a deployment trade-off (see Code Review Agents on Human-Verified Bugs (CRBugBench)).
Verification cost at the patch layer. The audit's 64% discard rate is the patch-side instance of the load Verification as the New Bottleneck describes: a noisy agent check moves work to the human who sorts its output, and the plugins' whole design aims to cut that sorting.

Connections#

Follow-Up Fixes on Agent PRs — the population rate for agent PRs that later need a fix. A PoC-driven hunt on security patches found a much higher share, which is one sign that the fix-PR construct is a floor
Code Review Agents on Human-Verified Bugs (CRBugBench) — bug-fix PRs are the hardest for reviewers (OR 0.521). That benchmark measures recall only, and this post measures precision only
Independent Assurance Plane — the framework the Replay and Prove agents partly implement: fresh-context evidence, a fail-closed verdict, a report with residual uncertainty. Here the patch author runs the whole pipeline
Optimizer–Evaluator Decoupling — fresh-context replay and a separate adjudicator as decoupling inside one vendor's pipeline, with no change of model
ExecCritic: Learn to Test, Test to Improve — the trained counterpart of the fail-to-pass check. ExecCritic qualifies a test by a clean failure on the base. The three-arm proof adds a revert arm that the patch alone must turn back into a failure
LLM-Driven Vulnerability Research — the discovery side of the same campaign. This page is what its fix PRs broke after merge
Loop Engineering — the same team's goal-prompt rules, and the shared lesson that minimal, opinionated guidance beats exhaustive instructions
Open Source Under Agent Contributions — maintainers writing THREAT_MODEL.md for patching agents, a maintainer-side answer to agent contribution volume
Human-Governed Skill Maintenance — the plugins ship in trailofbits/skills, one of the five repositories whose skill-maintenance history that page measures
Failures That Look Like Success — strict step schemas exist so each step checks the previous step's raw evidence, not its summary
Verification as the New Bottleneck — a 64% discard rate on agent regression candidates is verification load moved onto the engineer
Trail of Bits, Codex, OpenAI

Open Questions#

Does regression-hunt's false-positive reduction hold out of sample, and at what recall? The 25% (CI 14–38%) comes from the 21 regressions the plugin was designed from, against a one-line baseline, with borderline counts unpublished. Settler: per-arm confirmed / borderline / missed / false-positive counts on a held-out set of regressions introduced by bug-fix PRs, such as CRBugBench's SZZ-traced Fix track restricted to bug-fixing inducing PRs.
Do the three-arm proof and the mandatory regression hunt lower the merged-patch regression rate below the audit's ~16%? Trigger: Trail of Bits' announced patching-bugs evaluation, or the same full-merge regression review repeated on Patch the Planet PRs written with the plugins.

Sources#

Your AI agent fixed the bug, but what else did it change? — Anish Naik & Marcelo Morales (Trail of Bits), Your AI agent fixed the bug, but what else did it change?, Trail of Bits blog, 2026-10-21, ~1,800 words, case-study, tier kept (vendor launch, see the evidence note). Used throughout: the regression tally and definition, the 58-candidate precision, the four principles, both plugin pipelines, the 25% / 14–38% evaluation and its borderline cases, the planned work, and the two project files. The three figures are diagrams. Their text is transcribed in the raw's note blocks, and this page relies on the transcriptions.
How we use /goal to find bugs in Patch the Planet — Trail of Bits, 2026-07-28 (case-study): the campaign's bug-finding side, and the goal-prompt lesson cited in principle 1.