Sources#

• BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL
• COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
• Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning
• Staleness-Learning Rate Scaling Laws for Asynchronous RLHF

The Stale Advantage Without a Critic#

Question (from Anchored Bellman-Residual Correction (BRACE), #oq/now): BRACE is demonstrated only against a PPO critic, and its baselines never include GRPO. Does a k-capped Bellman-residual correction have any analogue for a critic-free, group-relative objective, or is the correction structurally tied to having a value function to regress at all?

Short answer. Half of it is tied to the critic and half is not. BRACE does two things at once. It fixes the critic's regression target, which needs a value function and has no group-relative counterpart. It also reweights the stale continuation inside the advantage, and that half survives removing the critic: with V ≡ 0, BRACE's own target reduces to a critic-free estimator that could weight a group-normalized advantage. No compiled source runs that estimator or any other correction on the group side, so whether the analogue works is open. Everything in §2–§4 below is wiki inference: algebra on the two papers' published estimators, not a claim either paper makes. §5 lists what is measured.

1. What a "stale advantage" is in each objective#

The question's premise is that BRACE's diagnosis (the uncorrected target's fixed point is V^µ) presupposes a value target. It does, but the diagnosis has two parts, and only one of them needs a critic.

• The continuation. BRACE and COPC both run at γ = λ = 1 with a terminal-only reward. There, uncorrected GAE gives Â_t = R − V(s_t). BRACE notes the target "collapses to the raw return R for every position" in this case (BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL §3.1, Anchored Bellman-Residual Correction (BRACE)). R is drawn from µ's continuation, so E_µ[R | s_t, a_t] = Q^µ(s_t, a_t), not Q^π. COPC names this advantage staleness: a token gets the right policy-side weight and is still credited with "behavior-policy continuations" (Coupled Off-Policy Correction (COPC), raw §3.2).
• The baseline. The V(s_t) that GAE subtracts is fit on µ-data and converges to V^µ (BRACE Proposition 1, COPC Proposition 1).

GRPO's advantage is (r_i − mean(r_{1:G})) / std(r_{1:G}) (Group Relative Policy Optimization (GRPO); Staleness-Learning Rate Scaling Laws for Asynchronous RLHF Eq. 1). Under asynchrony:

• The continuation part is identical in kind. r_i is the same µ-drawn return R, credited to every token of response i by a per-token ratio. At γ = λ = 1, uncorrected async PPO and uncorrected async GRPO carry the same continuation bias. Neither one corrects it.
• The baseline part is different. The group mean is a Monte-Carlo estimate of V^µ(x) at the prompt only, re-estimated from each fresh group. A critic carries a per-position V^µ(s_t) learned by regression across many stale batches. Under exact per-token ratios, any baseline that does not depend on the action contributes zero expected gradient (E_µ[ρ_t ∇log π_θ | s_t] = 0). So a stale baseline, per-prompt or per-state, mainly adds variance, not first-order bias. That stops holding exactly once clipping or masking enters, because the baseline's sign and size change which tokens are clipped.
• The std has no critic analogue. The normalizer is µ's reward spread for that prompt, so under staleness each prompt is weighted by 1/σ_µ(x) instead of 1/σ_π(x). That distorts how prompts are weighted against each other, not how tokens are credited. Error-Penalized Abstention Training shows the normalizer can rewrite the objective when the spread depends on the policy. That result is synchronous, but it is the mechanism a stale σ would act through.

The corpus does name this object. Song et al. write the behavior policy as an explicit argument of the GRPO surrogate, and say it captures "the importance ratio denominator, the clipping boundary induced by the rollout policy, and the group statistics of the sampled batch." Their Assumption 3 bounds how sensitive "group-normalized advantages" are to the behavior policy (Staleness-Learning Rate Scaling Laws for Asynchronous RLHF §2, §3.1). That dependence is folded into the O(Sη) bias bound on Staleness–Learning-Rate Scaling. It is bounded, not corrected, and its share of the bound is never separated out.

2. BRACE with the critic removed#

BRACE's target in its corrected-window-plus-anchored-return form (raw Eq. 26) is

V_targ(s) = V(s) + Σ_{t=s}^{j−1} (Π_{i=s}^{t−1} c_i) δ_t V + Π_k(s)[R − V(s_j)], with j = min(s+k, T), δ_t V = ρ_t(r_t + V(s_{t+1}) − V(s_t)), c_i = λ·min(c̄, w_i), and actor advantage Â_s = r_s + V_targ(s_{s+1}) − V(s_s) (Eq. 27).

Set V ≡ 0 and keep the terminal-only reward. Every window residual is then ρ_t r_t = 0 for t < T, the anchor's V(s_j) term vanishes, and

Â_s = Π_k(s+1) · R

That is the realized return times a k-capped product of truncated per-token ratios over the next k tokens. It is defined with no critic, and nothing stops R from being replaced by the group-normalized Â_i. So the k-cap does have an algebraic analogue for a group-relative objective: a forward, truncated importance weight on each token's advantage.

Three things are lost in the reduction, and they are why the analogue is untested rather than free:

1. Nothing is left to regress. BRACE's Propositions 2–3 (a unique fixed point, and window-endpoint displacement bounded by Θ_k‖g‖∞) are statements about a value function being fit. With V ≡ 0 they no longer describe anything. The regression-target half of BRACE is the part structurally tied to a critic.
2. The control variate is gone. In BRACE the capped product multiplies a residual centred by V. With V ≡ 0 it multiplies the raw (or group-normalized) return. BRACE's §C.1 noise argument and its k = 100, c̄ = 1.1 settings were derived and tuned for the centred case.
3. The product shrinks early credit. BRACE's own appendix shows E_µ[c_i | s_i] ≤ 1, and Jensen makes the log-product drift negative. Weighting Â_i by a 100-factor product would therefore systematically down-weight tokens whose next 100 tokens diverged from π. Whether that helps or starves the gradient is exactly the experiment nobody has run.

3. COPC with the critic removed: it degenerates#

The same substitution in COPC's advantage Â_t = δ_t + Σ_{j>t} ρ̄_j δ_j (Coupled Off-Policy Correction (COPC), raw §3.2) gives δ_j = r_j, which is zero except at T, so

Â_t = ρ̄_T · R for every t < T

That is one clipped ratio, in [0.80, 1.28], on the terminal token, applied to every token's credit. It says nothing about the continuation between t and T. COPC's per-token residual weighting does its correcting through the critic's intermediate residuals, so it is structurally tied to a value function in a way BRACE's capped product is not. This is a second, independent reason the two estimators differ, beside the per-token-noise tension recorded on Coupled Off-Policy Correction (COPC): with the critic gone, BRACE's product still carries the continuation, and COPC's single ratio does not.

4. What a correction on the group axis would correct, and what the corpus has#

From §1, a group-relative correction could act on three objects:

Correcting the group mean or std with importance weights needs the full-response ratio product, because the mean sits at position 0 and the whole response is its continuation. That is the V-trace coverage-versus-control problem BRACE §3.2 analyses, at full length and with no k-cap available, since there is no intermediate V(s_j) to anchor the tail on. So the baseline correction is the hardest of the three, and §1 argues it is also the one that matters least, because it is mostly variance.

What the corpus does have on the group side is actor-side only. DIS on GRPO rescues SAO's ~160-step collapse (Group Relative Policy Optimization (GRPO), Single-Rollout Optimization). ESTR's entropy-scaled keep mask runs on group-normalized advantages (Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning §4) and prices removing the correction entirely at 9.6 / 35.4 / 3.9 points against synchronous GRPO. Both correct how a token is weighted, ρ, and leave Â alone. SAO's other response is to delete the group and bring back a critic (Single-Rollout Optimization), which avoids the question rather than answering it. Nothing in the corpus recomputes a group baseline or reweights group statistics.

5. What is measured#

One source runs a group-relative arm next to critic-side corrections. In COPC's async harness, critic-free DAPO (group-standardized advantages, dynamic sampling, no group-side correction; raw Appendix C.2.1) scores:

• Uncorrected group beats uncorrected critic by 2.72 (math) and 3.89 (search). It also beats V-trace in both settings and SAO on search.
• The corrected critic beats the group by 3.94 and 3.28.
• DAPO never collapses over 450 search steps, while SAO, AReaL, V-trace and COPC's actor-only ablation do (Figure 2, chart read).

§1 offers a reading of the first point, and it is inference: both uncorrected arms carry the same continuation bias, so the gap between them is more plausibly the critic's learning variance and its cross-batch regression memory than a difference in staleness bias. COPC never tests this. DAPO also differs from the PPO arm in more than the critic (dynamic sampling, token-level loss averaging on math), the baselines are rival-implemented, each task runs on one 3–4B backbone, and staleness is 8 / 14. The numbers show that an uncorrected group is a stronger floor than an uncorrected critic in one harness. They say nothing about what a group-side correction would add.

What remains open#

• Does the V≡0 BRACE form, Π_k(s+1)·Â_i^{GRPO}, improve asynchronous GRPO over an actor-only keep rule, or does the uncentred product just starve early-token credit? No source runs it.
• Does importance-weighting or recomputing the group mean and std measurably change async GRPO? §1 predicts a variance effect, not a bias effect. No source tests either.
• Does the 3–4 point gap between corrected critic and uncorrected group survive a group arm that also carries a continuation correction? That is the comparison that would say whether the critic is needed or only the correction is.

All three need an experiment that is not in the vault, which makes the residue source-bound rather than synthesis-bound.

Sources drawn on#

• Anchored Bellman-Residual Correction (BRACE), BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL §3.1, §3.2, §4.1–4.3 (Eqs. 17, 20–27), Appendix C: the target algebra that §2 reduces.
• Coupled Off-Policy Correction (COPC), COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning §3.2 (Proposition 1, the corrected advantage), Tables 1–2, Figure 2, Appendix C.2.1 (the DAPO configuration).
• Group Relative Policy Optimization (GRPO): the group advantage, the DIS-on-GRPO rescue, ESTR's synchronous ceiling, the COPC pendulum reframing.
• Staleness–Learning-Rate Scaling, Staleness-Learning Rate Scaling Laws for Asynchronous RLHF §2–3.1: the behavior-policy-dependent surrogate, Assumption 3 naming group statistics.
• Asynchronous RL for LLMs, Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning: actor-side keep rules on group-normalized advantages.
• Single-Rollout Optimization: removing the group instead of correcting it.
• Error-Penalized Abstention Training: the group normalizer as part of the objective rather than an estimator.