Dual-Scope Agent PRs (Application Code Plus Deployment Infrastructure)

Sources#

• Co-Modifying Application Code and Deployment Infrastructure: A Causal Study of AI Agent Pull Requests

Summary#

A dual-scope PR changes application code and deployment infrastructure in the same pull request: a feature plus the CI workflow, Dockerfile or Terraform it needs in order to ship. Soualhi & Suleiman (Co-Modifying Application Code and Deployment Infrastructure: A Causal Study of AI Agent Pull Requests, arXiv 2610.12849, 2026-10-09, empirical) are the first in the vault to isolate this class and ask what it does to merge latency and human review. The design is a matched observational study over two public agent-PR corpora:

• AIDev v5 (the AIDev-pop subset, >100 stars): 21,855 agent PRs from six products (OpenAI Codex, Claude Code, GitHub Copilot, Devin, Cursor, Google Jules). It has no human arm.
• MOSAIC-3M, filtered to >100 stars: 2,570 agent PRs from five of the same products (no Cursor) plus 2,574 human PRs. This is the only human baseline in the study.
• Window: PRs created 2025-06-01 to 2025-07-31 and merged by 2025-08-31. Merged PRs only.

The headline result is a split, not a single effect. Reviewed dual-scope agent PRs stall, and unreviewed ones merge in minutes. The authors call this a "bimodal integration trap", and which mode a corpus shows decides whether its average latency penalty appears at all.

How a PR is classified#

There are no NLP task labels. AIDev's LLM task-type table stops working inside the window: valid coverage falls from 66.32% in July 2025 to 1.13% in August, and MOSAIC has no such table. Each changed file instead goes through a four-tier path classifier, and the first tier that matches wins:

1. Unambiguous infrastructure, in any directory: CI/CD (.github/workflows/), containers (Dockerfile) and declarative provisioning (.tf, cdk.json).
2. Tests (tests/, spec/ and filename patterns). This tier runs before tier 3, so test scripts and fixtures do not count as infrastructure.
3. Context-dependent infrastructure: shell and PowerShell scripts under scripts/ or tools/ (Automation), and terraform/-style directories (Provisioning).
4. Application source and application config. Everything else (docs, lockfiles, stray config) is auxiliary maintenance.

A PR is dual-scope if it has at least one application file, at least one infrastructure file, and at least two files. Pure-maintenance PRs are counted for prevalence and then excluded from the causal analysis.

Matching. Each dual-scope PR is paired 1:1, without replacement, with an application-only PR from the same exact stratum: same agent, same test-inclusion status, same repository archetype (eight classes, computed from the application files only so the treatment cannot shift it) and same language ecosystem (nine classes). Within the stratum the nearest PR on z-scored log application-file count is chosen, inside a 0.28 SD caliper. The caliper drops 29.82% of AIDev dual-scope PRs and 33.88% of MOSAIC's, leaving 687 and 80 pairs. The human baseline is matched the same way (79 pairs). Line churn and repository stars enter the regressions as covariates.

RQ1: rare, CI-heavy, and outnumbered by chores#

The application-only and infrastructure-only shares are from Fig. 1(a). The paper's text does not say whether its MOSAIC column covers agents only or the pooled cohort.

• Target subdomains (non-exclusive): CI/CD 77.12% / 71.79%, automation scripts 20.33% / 14.53%, containers 18.79% / 16.24%, provisioning 9.91% / 6.84%. The edits are small. The median dual-scope PR touches 2.0 infrastructure files (AIDev) or 1.0 (MOSAIC), 47.0% and 63.9% touch exactly one, and when only one is touched it is a CI manifest 74.1% and 65.9% of the time. Infrastructure files are actively authored, not just deleted, in 97.45% / 99.01% of these PRs.
• The "chore agent". Agent PRs with no application or infrastructure file at all (docs, tests, lockfiles) are 25.41% of AIDev and 17.70% of MOSAIC, four to six times the dual-scope share. README.md is the single most-modified file in them (44.70% and 25.02% of chore PRs). The authors argue this is LLM compute spent where Dependabot or Renovate would do. That is their opinion, not something the data measures.
• Vendor spread (Table 2, reconciled against pdftotext). Copilot and Devin have the highest dual-scope rates in both corpora (AIDev 15.44% and 12.12%), and Codex the lowest in both (1.87% and 1.80%). AIDev's pooled rate is therefore mostly a Codex rate, since Codex is 16,935 of 21,855 PRs. This is the Agent-Vendor Heterogeneity pattern again: the spread between products exceeds the pooled agent-vs-human gap (4.55% vs 3.30%).

A corpus warning that applies to every AIDev v5 file-level study. In the window, 18.66% of AIDev v5 PRs (5,015 of 26,870) have no file paths, because the GitHub API omits patch data and file arrays for very large PRs. 99.78% of those dropped PRs are Claude Code's, which is a 94.50% loss for that agent. Claude Code's AIDev dual-scope rate of 17.87% is therefore a survivorship artifact: the true value is bounded below at 0.98%, and 6.74% if MOSAIC's 6.09% is imputed. Across the whole AIDev v5 timeline the omission grows from 0.00% (January 2025) to over 30% (October 2025) and 65.64% (March 2026), 117,292 PRs in all. A longitudinal file-level series on AIDev v5 is unreliable for that reason alone. MOSAIC's file arrays are 99.88% complete after the authors drop 56 PRs whose file list is shorter than GitHub's changed_files count.

RQ2: the latency penalty, where it exists#

AIDev v5 (687 pairs). The medians are 3.47 h for application-only controls and 16.60 h for dual-scope treatments, a +13.13 h difference. The log-normal AFT model (pair-clustered SEs, adjusted for churn, test inclusion and stars) gives TR 3.237 [2.545, 4.116], p < 0.001. Size matching held: both arms have a median of 6.0 application files.

Read the headline carefully. +13.13 h is a difference of medians. The median paired difference is +0.84 h, and Cliff's δ is +0.202, a small effect (Wilcoxon p = 3.55 × 10⁻¹³). Most pairs differ little. The gap comes from the upper half of the treatment distribution, which is what the TR measures. The abstract also attributes the +13.13 h to "explicitly reviewed contributions", but the RQ2 analysis covers all matched pairs, reviewed or not. The reviewed/unreviewed split comes only in RQ3, as medians of 49.80 h (reviewed) vs 0.43 h (silent) in AIDev.

Which infrastructure drives it (Model 2, AIDev only, since MOSAIC's subdomain cells have N ≤ 12):

(Table 3, reconciled against pdftotext.) "Strict" dual-scope PRs, which touch core source, take a median 19.64 h. "Weak" ones, which touch only application config next to infrastructure, take 1.13 h, faster than the application-only baseline. The authors' mechanism: CI runs asynchronously on remote runners, agents have no loop that reads a remote build failure, and so a human ends up translating between the failed pipeline and the code. The mechanism is plausible, and the paper asserts it without measuring it (no CI-run or failure data is joined). An unmatched cut by directory is larger still: application-only PRs under src/ merge in a median 0.25 h, but 40.99 h when infrastructure is added (163.6×). That figure has no size control.

MOSAIC-3M (80 pairs): no penalty. TR 0.794 [0.377, 1.672], p = 0.544, medians 2.21 h treatment vs 2.93 h control. The paper names two causes and cannot separate them: low power, and the fact that half the treatment arm merges in the first 31 minutes (median 0.51 h for that half), with no human review.

RQ3: who gets reviewed#

"Silent integration" means merged with zero external human comments and zero external review iterations. Bot accounts are removed by a regex sanitizer (63.80% / 67.96% of raw comments are bots), and author self-comments are excluded. That exclusion moves 2.49% / 3.23% of PRs between categories.

Matched cohorts (Table 4, reconciled against pdftotext):

Formal review and silent merge do not sum to 100%. The remainder got comments without a formal review.

Three readings, in descending order of strength:

1. Agent PRs get less review than human PRs at matched size, whatever their scope. Even the application-only rows show it: 42.5% of MOSAIC agent PRs merge silently against 21.5% of human PRs, with both arms at a median of 4.0 application files. This is the most robust result in the paper.
2. Adding infrastructure does not move human-PR review (p = 0.839). Its effect on agent PRs has opposite signs in the two corpora: more review in AIDev (p = 0.015) and less in MOSAIC (p = 0.052, which misses 0.05; the asymptotic CI excludes 1, the exact test does not). The paper's own word is "volatile". The abstract's "43.1% to 56.2% bypass inspection" combines the AIDev row, where infrastructure raised review, with the MOSAIC row, where the shift is not significant. Both are silent shares, but they are not one effect.
3. The governance stratification (unmatched MOSAIC, all scopes, not dual-scope only). Core maintainers' PRs are reviewed less (50.4%) than external contributors' (71.4%). Within Stratum A, 2,232 PRs whose authors have no repository association, human PRs are reviewed 82.7% of the time and agent PRs 63.6%. Adjusted for volume, churn, tests and stars, OR = 0.363 [0.293, 0.451]. The "63.7% lower odds of review" is that odds ratio, not a 63.7% drop in the review rate. The rate gap is 19.1 points. Note that this figure is not about dual-scope PRs. The paper places it next to the dual-scope silent-merge share, but the two come from different analyses.

The confound the stratification does not close. Stratum A is defined by GitHub's author-association field on the PR. Copilot's coding agent, Devin and Jules open PRs from bot or app accounts, and those accounts can show no association even when a maintainer with write access assigned the task and will merge the result. Codex is the opposite case: its PRs are opened under the invoking developer's own login. So "an agent PR from an unprivileged external contributor" may often be a maintainer's own delegated work, merged by that maintainer. In that case it is review-skipping by the dispatcher, not a stranger getting through the gate. This matters for the supply-chain reading below, and the paper does not test it. The paper does identify who drives the silent pool: Codex accounts for 90.2% of AIDev's silent merges, and Claude for 31.4% of MOSAIC's. Over all agent PRs the silent share is 79.27% (AIDev) and 49.01% (MOSAIC). For Codex, author-self exclusion means a developer merging their own Codex PR counts as silent by construction.

Where review effort goes. CI manifests cause the longest stalls, but automation scripts draw the most discussion: 3.11 review iterations and 4.24 comments per AIDev dual-scope PR, against 2.22 and 2.93 for CI. In MOSAIC the figures are 3.42 iterations for scripts against 0.88 for CI. Procedural code gets read, and declarative pipeline config mostly waits.

The circular-verification argument and the prescription#

An unreviewed PR that edits .github/workflows/ also edits the check that validates it, so a green status on that PR "provide[s] zero independent assurance of build integrity." The authors map this to OWASP's Poisoned Pipeline Execution (CI/CD Top 10). The argument is sound for the pull_request trigger, which runs the PR's own workflow definitions. It is an argument, and the paper measures no compromise, malicious edit or broken build. What it measures is the base rate of the precondition: CI-touching agent PRs that no external human commented on or reviewed.

Prescriptions (§4.3), none tested in the paper:

• Repository side: path-based CODEOWNERS plus rulesets that disable auto-merge and require one non-author human approval whenever a PR touches .github/workflows/**, terraform/** or k8s/**. The authors claim this costs application-only PRs nothing.
• Agent side: shift pipeline verification left, with local CI dry-runs (act), container builds, and manifest linters (actionlint, hadolint) before the PR opens.
• Benchmark side: deployment-aware evaluation, in which a patch counts as resolved only if it also passes manifest analysis and an ephemeral runner job. The authors set this against SWE-bench, DevOps-Gym, IaC-Eval and DevOps-Bench, which all grade in sandboxes.

How it lands on the gate designs in the vault. Risk-Tiered Auto-Approval already derived the path prescription from Security Debt of Agent-Generated Code's finding that 87.6% of security smells sit in Actions workflows and Dockerfiles. This paper supplies the other half of that argument. CI paths are where agent PRs go when they touch infrastructure (72–77%), and in MOSAIC the matched dual-scope agent PRs are the ones most likely to skip review. But it also prices the gate's cost: in AIDev, CI-touching PRs already take 3.8× as long to merge, so a required approval lands on the slowest queue. The author's claim that the gate costs application-only PRs nothing holds by construction, since those PRs never touch the paths. The other half of the cost is the extra wait on dual-scope PRs, and nobody has measured it.

Evidence and scope#

• empirical, observational, with matching under an exchangeability assumption. Latent task intent (fix vs feature) is an admitted unobserved confounder. Only merged PRs are studied, so abandoned dual-scope PRs are invisible, and the authors call their latency figures a lower bound.
• Two academic authors (an independent researcher and UNSW), no product and no declared funding. An FSE 2027 submission, given the replication-package URL; not yet peer-reviewed. The replication package is anonymized now and promised to Zenodo on acceptance.
• The MOSAIC arms are small: 80 agent pairs and 79 human pairs. Every MOSAIC dual-scope result has wide intervals, and Model 2 could not be run on MOSAIC at all.
• Population: public repositories with >100 stars, created June–July 2025. The authors exclude enterprise monorepos with SRE gates, and they predict that event-driven agents which read CI failures will change the dynamics.
• Internal consistency: Table 1 and the prose agree on the funnel. The "56 dropped" and "6 of the remaining 5,150" payload-safeguard counts in §2.2 reconcile against Table 1's 62 omissions.

Connections#

• Risk-Tiered Auto-Approval — the gate design this paper's prescription is a version of. CI, container and IaC paths on the deny-list, here with a measured base rate of unreviewed infrastructure-touching agent merges and a measured latency cost on the paths the gate would hold. The page's open question about whether path entries pay for themselves gets its cost half, partially
• Security Debt of Agent-Generated Code — same surface, different quantity. That page measures what is wrong in agent CI and Dockerfile diffs (38.9% smell prevalence, 87.6% of it in those files). This one measures how often such diffs get looked at, and adds the first agent-vs-human volume figure for infrastructure-touching PRs (4.55% vs 3.30% in MOSAIC)
• Review as the Control Point — a share-denominated no-review measurement split by authorship at matched size, which the page's convergence question asks for. It covers only one window (June–July 2025), so it fixes one point on the curve and not the trend. Agent silent share is 42.5% against 21.5% for humans on matched MOSAIC application-only PRs
• Reviewer Habituation on Agent Pull Requests — a second mechanism for agent PRs getting waved through, and a different one. That page has the same reviewer's approval rising with exposure. Here, author attribution alone lowers the odds of any review at all (OR 0.363 among unassociated authors). One is about the reviewer, the other about the gate
• Review Wait Under AI Code Volume — another review-time quantity for that page's table. Time-to-merge on size-matched pairs, with a corpus-dependent sign: TR 3.24 in AIDev, null in MOSAIC because half the treatment arm skips review
• Agent-Vendor Heterogeneity — the vendor spread reappears (Copilot and Devin highest dual-scope rates, Codex lowest in both corpora). This paper also supplies the AIDev v5 file-array omission audit (94.50% of Claude Code PRs in the window lost, 65.64% of all PRs by March 2026), which any AIDev file-level vendor comparison has to carry
• Structural Artifact Monitoring — the IaC pre-merge monitor, and why it has to sit outside the PR's reach. Its proposed fix for the cdk.out bypass is a CI step that re-runs the build. The CI definition is exactly the file class dual-scope agent PRs edit 72–77% of the time, and in MOSAIC 56.2% of those PRs go unreviewed. A check that the PR under test can rewrite is the circular verification this paper names
• Agent Supply Chain Risk — Poisoned Pipeline Execution as the named attack class. This paper is the base rate of its precondition among agent PRs, not evidence of an attack
• Open Source Under Agent Contributions — the maintainer-gate version: among authors with no repository association, agent PRs get reviewed 63.6% of the time against 82.7% for humans. The bot-account dispatcher confound above decides whether that is strangers getting through or maintainers skipping review of their own delegated work
• Follow-Up Fixes on Agent PRs — the post-merge cost on the same AIDev-family population, and the outcome this paper lacks. Nothing here says whether silent dual-scope merges break builds or draw fixes more often than reviewed ones
• Verification as the New Bottleneck (hub) — the CI-manifest stall is that bottleneck in a specific place: the human acting as the translation layer between a remote build failure and the code

Open Questions#

• Is the agent review deficit about the author or about the dispatcher? Stratum A's "no repository association" is read off the PR's author account, and bot-account agents (Copilot, Devin, Jules) can show no association even when a write-access maintainer assigned and merges the work. Re-stratify on the dispatching or merging user (assignee, trigger event, merger's permission) instead of the PR author. If the OR = 0.363 deficit vanishes for maintainer-dispatched PRs, the supply-chain reading becomes a self-review reading. MOSAIC's replication package plus the GitHub timeline API is enough.
• Do path-based review rules on infrastructure paths remove silent dual-scope merges without lengthening the merge queue? The prescription is untested, and AIDev already shows CI-touching PRs at 3.8× merge time. Settler: a before/after on repositories that adopted CODEOWNERS or rulesets for .github/workflows/** during the agent era, comparing silent-merge share and time-to-merge for dual-scope agent PRs, or any CI-run outcome joined to the reviewed vs silent split.

Sources#

• Co-Modifying Application Code and Deployment Infrastructure: A Causal Study of AI Agent Pull Requests — Mohammad Wathiq Soualhi (independent, Malaysia) & Basem Suleiman (UNSW), Co-Modifying Application Code and Deployment Infrastructure: A Causal Study of AI Agent Pull Requests, arXiv 2610.12849, 2026-10-09, 21pp, empirical. Cited: abstract; §2.1–2.2 + Table 1 (datasets, window, file-array omission audit); §2.3 (four-tier taxonomy, dual-scope definition, task-label decay); §2.4 + Algorithm 1 (exact-strata caliper matching, caliper pruning, AFT/McNemar/logistic models, silent-integration definition); §3.1 + Fig. 1 + Table 2 (prevalence, subdomains, chore PRs, Claude Code survivorship); §3.2 + Fig. 2 + Table 3 (latency, strict vs weak, subdomain TRs, directory multipliers); §3.3 + Table 4 (intervention bifurcation, Stratum A, review volume by subdomain, reviewed vs silent latency); §4 (circular verification, PPE, prescriptions); §5 (threats). Parse status: clean for every table cited. Tables 2, 3 and 4 were reconciled cell by cell against pdftotext -layout on the local PDF, and all match. Table 2's "No Baseline" in the Dual-Scope column of the Human Baseline row is a merged cell spanning the three AIDev columns in the PDF, so the verify table-shift flag on it is a false positive. Table 3's first row welds the "Model 1" group label onto the AIDev row, which is cosmetic. The docling body lists the RQs out of order (RQ1, RQ3, RQ2) and splits two paragraphs mid-sentence around Algorithm 1 and §1. These are parse ordering artifacts, and the text is complete. Figures 1 and 2 were not opened, because every figure number used is stated in the prose