Single-Rollout Optimization

Sources#

• Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning
• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
• TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

Summary#

SAO (Single-rollout Asynchronous Optimization) replaces GRPO's group-of-responses-per-prompt with one rollout per prompt (group size 1), fed to training the instant it completes. Two payoffs: it cuts off-policy drift (no waiting on the slowest group member — see Asynchronous RL for LLMs), and it fits the settings where GRPO structurally cannot — online and complex agentic environments that only ever return a single trajectory of feedback per prompt.

The catch is the reason the field abandoned single-trajectory methods in the first place: variance. With no group, there is no group-relative baseline for advantage estimation, so single-rollout gradients are as noisy as REINFORCE. SAO's answer is a deliberate counter-current — it re-embraces the value model (critic) the recent GRPO/RLOO wave was built to avoid, and spends its entire engineering budget making that critic stable enough to serve as the baseline. Deployed to train GLM-5.2 (750B-A40B).

The counter-current worth naming#

For two years the RL-for-LLM direction of travel was away from value functions. GRPO, RLOO and friends sell themselves on being critic-free: no separate value network to train, half the memory, none of the value-learning instability. SAO argues this is a dead end for the setting that now matters most. Critic-free advantage estimation is structurally dependent on a group — you need multiple responses to the same prompt to compute a relative reward — and asynchronous/online agentic feedback gives you exactly one. So the value model comes back, not out of nostalgia but because it is the only baseline that works from a single trajectory. The bet: a well-trained critic beats a group-relative baseline, and the instabilities that drove people off critics are fixable. Most of the paper is that fix.

The four things that make a single-rollout critic stable#

1. Faster value update than policy (TTUR, K=2). The core instability in single-rollout RL is the policy↔value interdependence: an inaccurate V_ϕ yields noisy advantages, which drive destructive policy updates. SAO decouples the update frequencies — K value-network updates per policy update (K=2 in the experiments) — so value estimates track the current policy before being used for advantage computation. This is a two-timescale update rule adapted for LLMs.

2. Frozen-attention critic. In pilot runs the value model's gradient norms were far larger than the policy's, and decomposition traced the instability to the full-attention layers, while the MoE layers stayed stable. So during RL, SAO freezes the attention modules of V_ϕ and trains only the MoE projections — the hypothesis being that pre-trained attention already attends to the right tokens, so restricting optimization to the MoE layers regularizes the critic. Ablation: removing this drops AIME2025 97.3 → 90.6.

3. Skip-observation token-level GAE. Agentic trajectories interleave model actions and environment feedback: T = [a₀, o₀, a₁, o₁, …]. Standard GAE computes value differences between adjacent tokens — but the boundary from an action's last token to an observation's first token is discontinuous from the model's perspective (the model didn't generate o_i), so estimating advantage across it makes the critic try to predict the value of an external environment state, injecting noise. SAO's fix bridges action→action directly, skipping observation tokens:

Â(a_{i,N}) = δ + γλ · Â(a_{i+1,0}), with δ = r_t + γ V(a_{i+1,0}) − V(a_{i,N})

Advantage estimation is thereby constrained to model-generated tokens only. Token-level beats step-level: treating each turn as one action (step-level GAE) underperforms — Table 5 at 400 steps: step-average 85.8, step-last-token 87.3, token-level 89.8 on AIME2025 — because finer-grained supervision captures logical transitions within a trajectory that a per-turn signal smooths away.

4. Scaled value pretraining. The critic's cold-start is a real bottleneck; scaling the value-pretraining corpus gives a robust initialization that lets the single-rollout and TTUR mechanisms work from early training rather than fighting a bad critic for hundreds of steps.

Results#

Every design choice is load-bearing — the ablation (Table 4) shows each removal costs accuracy, and the two cheaper single-rollout baselines (a running-mean-reward baseline, and vanilla VAPO) both trail badly or collapse:

Variant	AIME2025	BeyondAIME
SAO	97.3	74.8
w/o faster value	95.0	69.8
w/o frozen attention	90.6	74.5
Vanilla VAPO (no DIS)	91.3	69.0
Running-mean baseline	79.8	55.3

On coding, SWE-Bench Verified (Qwen3-30B-A3B backbone, OpenHands scaffold, 300 turns, 128k context): base 23.0 → GRPO+DIS 27.0 → SAO 29.8. On the four math-reasoning benchmarks in a reasoning-with-Python (TIR) setting, SAO beats both the SFT baseline and GRPO across the board (AIME2025 97.3, BeyondAIME 74.8, HMMT 88.3, IMOAnswerBench 74.0), landing near the much larger GLM-4.7.

The online-learning result: where single-rollout is uniquely suited#

SAO's sharpest claim is not a benchmark bump but a capability GRPO cannot have. In a non-stationary environment — feedback is one trajectory per prompt, and the reward criterion itself changes over time — GRPO's group-relative baseline is structurally inapplicable. SAO's value-based critic is not: it provides a state-dependent baseline from a single trajectory.

The demonstration is a simulated online writing task where the target style is switched mid-training between archetypes (cute, chuunibyou, classical), with GLM-4.7 as an LLM judge scoring r = r_quality × r_style ∈ {0,1}. When the preference shifts, SAO rapidly realigns — suppressing the old dominant style and converging on the new target — while a running-mean baseline (128-reward sliding window) lags, because its historical window stays biased toward the previous distribution. The critic tracks the shift; the running average can only average over it.

A rival stabilization design, and what it does not contest#

SAO's two contributions are separable (DIS buys stability, single-rollout buys the ceiling), which means a competing stabilizer attacks only the first. ESTR (Baidu et al., arXiv 2607.22186, 2026-07-24, empirical) is that competitor: it replaces the fixed-magnitude keep rule with a boundary scaled by each token's local entropy, |δ_t| ≤ √(τ(H_t + ε)), on the argument that the importance ratio's natural scale grows with entropy, so no position-invariant bound can separate amplified low-entropy sampling noise from legitimate high-entropy exploration. Its matched-budget ablation is the load-bearing one — recalibrate three keep rules to an identical 0.07% step-0 masked fraction and two still collapse, one of them while holding a budget of the winner's order — so what decides stability is the mean entropy of the masked population (0.11 vs 1.67 vs 3.47), not its size. Full treatment on Asynchronous RL for LLMs.

Three things are worth being precise about, because the overlap is narrower than it looks.

It does not touch the single-rollout half at all. ESTR runs GRPO advantages with n = 8 rollout samples per prompt on every task (Table 3) — a group, not a single rollout — and never discusses critics, value pretraining, or single-trajectory feedback. Nothing here bears on this page's central bet that a well-trained critic beats a group baseline.

It agrees with DIS on the behavior-proxy question. ESTR's Appendix G specifies "rollout log-probabilities as the proximal anchor (bypass mode)" — the same drop-π_θ_old move DIS makes, arrived at independently. Both papers also reject the staleness-decoupling family for the same reason: with a trajectory spanning several weight versions there is no single behavior policy to reconstruct. The live disagreement is narrow: the shape of the trust boundary, nothing else.

Whether the critique lands on DIS is a wiki inference, not ESTR's claim. ESTR never mentions DIS or SAO; its named baselines are IcePop and KPop. The bridge is this wiki's own note that DIS is "a stricter version of the IcePop mechanism" — and if that holds, being stricter makes the miscalibration worse rather than better, since narrowing a fixed interval cannot change its shape. Unconfirmed: SAO reports a clip ratio but never a masked-token entropy, which is exactly the diagnostic that separates the two designs. See the same discussion on Asynchronous RL for LLMs for what would settle it.

A design that bets the opposite way on the observation boundary#

Of SAO's four critic-stabilization moves, skip-observation GAE is the one that makes an implicit claim about the world rather than about optimization: that the action→observation boundary carries no learnable value signal worth propagating, only noise from an external state the model did not generate. TRACE (Tao et al., UW–Madison + Microsoft Research, arXiv 2607.13988, 2026-07-15, empirical) is the first source in the corpus to build a method on the exact opposite premise: it splits rollouts at tool-call boundaries and derives every one of its dense rewards from the value change across an observation, measured as the TD difference in a log-ratio gold-answer-predictability value read off a frozen reference model — no trained critic anywhere. On closed-web BrowseComp-Plus that dense boundary signal is worth 4.5 average points over outcome-only GRPO on a 4B backbone and 5.6 on a 30B-A3B (full treatment, numbers and parse notes on Turn-Level Credit Assignment).

Read this as analogy-strength evidence, not as an ablation of SAO's design. The distinction is load-bearing and easy to lose:

• The value functions are different in kind. SAO's concern is that a learned V_ϕ asked to estimate advantage across an observation ends up trying to predict an external environment state, and that error then feeds back into policy updates. TRACE's value is never learned and never optimized — it is a frozen copy of the policy initialization scoring gold-answer log-probability, differenced against itself. It cannot degrade the way a trained critic can, so its success at the boundary is not evidence that a critic would survive there.
• The two designs actually agree on the loss. TRACE masks tool observations out of the policy-gradient loss exactly as SAO does. So the live disagreement is not "train on observation tokens or not" — it is narrower: is the value difference across an observation signal or noise?
• No shared experiment exists. TRACE never mentions skip-observation GAE, runs no GAE arm at all (its controlled baselines are GRPO, GSPO and GiGRPO), and differs from SAO in task, backbone, objective and reward. What would settle it is skip-observation versus cross-observation GAE, same critic, same task.
• What it does move: it makes the premise contestable rather than assumed. On long-horizon search the observation is demonstrably what moves the value, and the movement is concentrated — in one of TRACE's qualitative traces the browser.open supplying the decisive phrase earns δ = +5.86 and the literal find confirming it on the next turn earns +0.00.

Connections#

• Asynchronous RL for LLMs — the other half of SAO; single-rollout is what removes the group synchronization barrier that async exposes, and DIS is what keeps the resulting updates stable — and the home of ESTR, the rival stabilizer that contests DIS's fixed-magnitude boundary without touching the single-rollout half
• Turn-Level Credit Assignment — the design that bets the other way on SAO's observation boundary: all of its reward signal is the value change across a tool observation, from a frozen probe rather than a trained critic
• Group Relative Policy Optimization (GRPO) — the method SAO replaces; the counter-current here is precisely a move back toward the critic GRPO removed
• LLM-as-a-Judge — the online-learning reward signal is an LLM judge (GLM-4.7) scoring quality × style
• Large-Scale Test-Time Compute — the long-horizon agentic models this trains are the ones whose capability then scales with inference budget
• The Open-Weight Frontier Gap — GLM-5.2 (750B-A40B), SAO's deployment target, is the frontier-open MoE that page tracks
• The Bitter Lesson — SAO both removes structure (drops the group baseline, drops π_θ_old) and adds a great deal (frozen-attention critic, skip-observation GAE, length-adaptive λ) — the same both-directions shape as Gemma 4
• Reward Hacking — a value-based critic gives a denser, state-dependent reward signal than a sparse group-relative one; whether that changes the Goodhart surface is unexplored
• GLM (Z.AI) — the production deployment: GLM-5.2 (750B-A40B), and GLM-4.7 as both a benchmark ceiling and the online-sim judge
• RL from Execution Feedback (RLEF) — the execution-feedback case this page's open question names, trained end to end: RLEF feeds the interpreter's failure text back across turns and rewards the survivor, so on code at least the environment response is treated as the load-bearing signal. It runs no GAE arm and its credit is coarser than SAO's, so it sharpens the doubt without resolving it
• Offline Multi-Step Tool-Use RL (SWiRL) — the systems problem sidestepped rather than solved: SWiRL never touches an environment during RL, scoring proposed actions against frozen context collected offline, so it trades this page's on-policy rollout engineering for an off-policy mismatch that nobody has priced at longer horizons

Open Questions#

• The whole method is a bet that a well-trained critic beats a group baseline. It wins here, on a 30B-A3B backbone with scaled value pretraining — but the critic doubles training memory. At what scale does the group-free simplicity of GRPO win back on cost even if it loses on quality?
• Frozen-attention is justified by a hypothesis ("pre-trained attention already attends to the right tokens"), validated only by the gradient-norm trace and one ablation. Does it hold when the value model must attend to tool outputs it never saw in pretraining?
• Skip-observation GAE assumes environment feedback carries no learnable value signal worth propagating. For agents where the environment response is the crucial information (a compiler error, a test result), is skipping it leaving signal on the table? Partially answered (2026-08-12), and only by analogy — the premise is now contestable, not refuted: TRACE builds an entire dense-reward method out of the value change across tool observations at exactly these boundaries, and on long-horizon search it is worth 4.5 (Qwen3-4B) and 5.6 (Qwen3-30B-A3B) average points over outcome-only GRPO on the same backbone, data and protocol — with the credit demonstrably concentrated on the observations that carry evidence (a decisive page-open earns δ = +5.86, the literal find confirming it on the next turn +0.00). So on at least one long-horizon environment the observation is not value-neutral. Three things keep it from settling the question: TRACE's value function is a frozen reference model's gold-answer log-probability, never trained, so it is immune to the specific failure this design choice defends a learned critic against; TRACE masks observation tokens from the loss exactly as SAO does, so the disagreement is only about the value difference across the boundary, not about training on it; and no shared experiment exists — TRACE runs no GAE arm and never mentions skip-observation, so the direct test (skip-observation vs cross-observation GAE, same critic, same task) remains unrun. The compiler-error / test-result case in the question is still untouched: this is retrieval, not execution feedback.
• The online-learning win is on a controlled simulated preference shift with an LLM judge. Real user-facing online adaptation — the paper flags this itself — needs safeguards, monitoring, and privacy review the study doesn't attempt.

Sources#

• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning — Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning, Hou, Li, Tang, Dong (Tsinghua / Z.AI), arXiv 2607.07508, 2026-07-08. §3.2 (single-rollout + value-model designs), §4 (results, ablations), §4.5 (online learning), Appendix A (step-vs-token GAE). empirical.
• Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning — Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning, Zhao, Xie, Zheng et al. (BUPT / Peking / USTC / CAS + Baidu), arXiv 2607.22186, v1 2026-07-24, empirical. §4.3 (the keep rule), §I.2 + Table 7 (matched-budget ablation), Appendix G + Table 3 (n = 8 rollout samples, "bypass mode" proximal anchor — the two facts that bound the overlap with this page). Full treatment and all four parse warnings on Asynchronous RL for LLMs.
• TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents — TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents, Tao, Peng, Yao, Ge, Cheng, Wang, Gao, Li (UW–Madison + Microsoft Research), arXiv 2607.13988, v1 2026-07-15, empirical. §3.1 (tool-boundary states; observation tokens masked from the loss, as here), §3.2–3.3 (the frozen-reference log-ratio value and its TD credit — the mechanism that differs in kind from a trained critic), §4.1 (the controlled baseline set, which contains no GAE arm), §A.5 (the concentrated per-turn credit traces). Full treatment, all tables and all parse notes on Turn-Level Credit Assignment.