Coupled Off-Policy Correction (COPC)

Sources#

• COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

Summary#

The asynchronous-RL stabilizers on Asynchronous RL for LLMs split into two camps that had never been run together. The actor-side keep rules (DIS, ESTR, IcePop, KPop, AReaL) decide how much a stale token's gradient counts. BRACE corrects the critic's regression target and leaves the actor alone. COPC (Hu, Zhou, Zhang et al., Fudan / Meituan / East China Normal, arXiv 2610.09597, 2026-10-07, empirical) runs both in one loop and studies how they interact.

Its starting claim, advantage staleness: a token can get the right policy-side weight and still be credited with the wrong advantage, because the advantage is built from continuations the stale behavior policy µ generated. Policy-side correction cannot reach that error. Its main result: the two corrections' errors are not separable, so their strengths must be chosen together rather than tuned one at a time.

Two channels, one actor update#

The paper writes the actor update in a general two-channel form, g(η_P, η_A) = E[(1/|y|) Σ_t w_t(η_P) ψ_t(θ) Â_t(η_A)], and measures it against an idealized reference g* that uses the raw ratio ρ_t and the true current-policy advantage A^π_t.

• Policy channel, w_t(η_P). COPC adopts SAO's DIS mask unchanged: w_t = ρ_t · 1{ℓ_P < ρ_t < u_P}. The ratio's gradient path is kept inside the interval, and the token is zeroed outside it.
• Advantage channel, Â_t(η_A). Each TD residual is reweighted by its own two-sided clipped ratio ρ̄_j = clip(ρ_j, ℓ_A, u_A), giving the corrected return v̂_t = V_t + Σ_{j≥t} (γλ)^{j−t} ρ̄_j δ_j and advantage Â_t = δ_t + Σ_{j>t} (γλ)^{j−t} ρ̄_j δ_j. The corrected returns are the critic's target (clipped value regression), and the same quantities build the actor's advantage. Masked tokens still contribute their residuals to return estimation and critic training. The mask acts only on the actor gradient.

Both intervals come from shared base margins, ℓ_X = 1 − η_X ε_ℓ, u_X = 1 + η_X ε_h, so a smaller η means stronger correction. Main runs use ε_ℓ = 0.4, ε_h = 0.56, (η_P, η_A) = (0.5, 0.5), which gives [0.80, 1.28] for both channels. Setting η_A = 0 recovers plain GAE, and that is the actor-only ablation "COPC w/o RC".

The theory: why the channels can't be tuned separately#

Proposition 1 (fixed point). The expected corrected-return operator has the unique fixed point V^π̃, where π̃(a|s) ∝ µ(a|s) ρ̄(s,a). With no advantage-side correction (ρ̄ ≡ 1), π̃ = µ: the return target stays tied to the stale policy, however the actor is weighted. With unclipped ratios, π̃ = π_θ, which is exact but high-variance. Two-sided clipping sits between the two, so COPC's critic deliberately targets a biased intermediate policy, not V^π. This is the same convergence-to-V^µ result BRACE proves as its own Proposition 1. Two groups reached it independently within a month: COPC does not cite BRACE, though it does cite ESTR.

Proposition 2 (bias coupling). Write Δ_t = w_t − ρ_t for the policy-weight error and e_t = Â_t − A^π_t for the advantage error. The update bias is then exactly E[(1/|y|) Σ ψ_t (Δ_t A^π_t + ρ_t e_t + Δ_t e_t)]. The third term is a multiplicative interaction that can reinforce or offset the two direct terms, so how much one channel reduces bias depends on the other channel's residual error.

Proposition 3 (variance coupling). Each token's gradient variance contains E[w_t² ‖ψ_t‖² σ_t²(η_A) / |y|²], so squared policy weights scale the advantage's conditional uncertainty. Loosening the mask amplifies whatever noise the advantage channel leaves behind. Corollary 1 combines the two into a gradient MSE, ‖g − g*‖² + V[Z̄_K]/N, which need not be separable in (η_P, η_A).

The analysis gives no optimal coordination rule. What it supports is a directional coordination hypothesis: stronger correction in one channel should favor weaker correction in the other.

The sweep that tests it#

Figure 1 sweeps a 3×3 grid, η_P, η_A ∈ {0.25, 0.5, 0.75}, at staleness s ∈ {8, 12, 16} on AIME 2025 avg@32 (Qwen3-4B). The cell values are printed on the heatmap and are not chart reads:

• The reversal. At η_P = 0.25, raising η_A from 0.25 to 0.75 gains about 4 points at every staleness level. At η_P = 0.75, the same move loses about 2–3.5 points. The sign of one knob's effect depends on the other knob.
• Over-correction is a real failure mode. Making both channels strong is the worst corner at every staleness level, about 5 points below the center. Applying both corrections at full strength does not add up to more stability. It does worse than either balanced corner.
• The optimum is broad. The center and both off-diagonal corners all land within about 1–2 points of the best cell, so the one shared setting is used for every task with no per-staleness retuning.

One selection caveat the paper does not raise. The shipped (0.5, 0.5) is the sweep's best cell at s = 8 and s = 16, and the sweep metric (AIME 2025 avg@32) is also a Table 1 test column. The s = 8 center cell (53.58) is identical to COPC's AIME25 entry in Table 1. So for that column the shared setting was picked on the number it then reports. It is also the grid's natural midpoint, and the other four Table 1 benchmarks were not in the sweep, so the leak is narrow. But it is there.

Results#

Two settings, each on one backbone only. Tool-integrated math uses Qwen3-4B-Instruct after ReTool SFT, with RL on DAPO-Math-17K and avg@32 reported. Search uses Llama-3.2-3B-Instruct after a 2,000-example SFT, with a Search-R1 setup and greedy exact-match reported. Training runs on Slime on 8 H200s with a VAPO-strengthened PPO recipe and γ = λ = 1. Max staleness is 8 for math and 14 for search, at actor LR 1×10⁻⁶. Tables 1–3 were reconciled cell for cell against pdftotext -layout.

• Headline margins. COPC beats the strongest async baseline (AReaL in both settings) by 2.03 on math and 2.77 on search, and beats PPO by 6.65 and 7.17. It wins all five math benchmarks. On search it wins five of seven: it loses PopQA to its own ablation (38.80 against 39.80) and 2Wiki to AReaL (40.40 against 41.00). The abstract's "highest reported performance" is a claim about averages.
• The ablation is the load-bearing comparison. Adding the advantage channel to an identical masked actor raises math from 48.11 to 50.29 and search from 40.80 to 44.57. It beats the ablation on all five math benchmarks and six of seven search benchmarks.
• V-trace, the classical critic-side fix, is the weakest correction in both settings. Run with ρ̄ = c̄ = 1 truncation it barely clears uncorrected PPO, and it is the first method to collapse on search. That independently corroborates BRACE's claim that V-trace does not transfer to long-horizon LLM RL.

Stability (Figure 2 left, search training score; all step positions are approximate chart reads). COPC holds near 0.52 for all 450 steps. V-trace collapses first, at around step 270. COPC w/o RC follows at around step 370, and SAO and AReaL at around step 410, all dropping to roughly 0.1–0.2. Uncorrected async PPO and critic-free DAPO do not collapse: they plateau near 0.48–0.50. So the paper's "most evaluated asynchronous baselines exhibit late-stage performance collapse" means three of five baselines, plus the ablation. Every collapser has an actor-side correction, a critic, or both, while the two non-collapsers have no explicit off-policy correction at all. Which checkpoint the tables report is not stated.

Staleness (Figure 2 right, AIME 2025 avg@32, chart reads). Curves at s ∈ {8, 12, 16, 32, 64} all climb from about 24 to about 52–55 over 120 steps. s = 64 ends a little lowest, and the paper calls final performance "comparable across all settings". That is a wider staleness range than ESTR's 30 or BRACE's 50, but over a training horizon only 120 steps long.

Cost (Table 3). On math, synchronous PPO takes 410.59 s/step at 614.96 tok/s/GPU, async PPO 236.80 s at 856.93 (1.73×), and COPC 238.95 s at 836.85 (1.72×). On search the same three are 213.18 s at 65.35, 124.90 s at 96.44 (1.71×), and 126.18 s at 94.92 (1.69×). COPC adds 0.9–1.0% step time and costs 1.6–2.3% throughput against async PPO. Unlike BRACE's Table 2, where a 1.5% step-time cost sits beside an unexplained 8.7% throughput drop, the two numbers here agree. The 1.7× speedup is measured at max staleness 8/14, not at 64.

How it relates to BRACE#

COPC and BRACE make the same diagnosis and build different estimators.

• Same diagnosis. Both prove that an uncorrected critic converges to V^µ. Both find V-trace inadequate: BRACE argues it analytically, and COPC's V-trace baseline is its weakest arm.
• Different estimator. BRACE forms a per-residual product of importance ratios, caps it at k = 100 tokens, and gives the tail a constant weight of 1. COPC never forms a product. Each residual carries only its own clipped single-token ratio, along the whole response. (Wiki inference, not either paper's claim: that makes COPC's estimator a V-trace variant whose trace coefficients are a constant λ = 1 and whose per-residual weight is clipped two-sided rather than min(ρ̄, ·). It escapes the product-length problem BRACE's k-cap targets by having no product to bound.)
• Tension. BRACE's Appendix C.1 argues that per-token weights in the region its constant tail covers break telescoping and inject noise that grows like O(√(T−s−k)). COPC applies exactly such per-token weights everywhere, and relies on the narrow [0.80, 1.28] clip to bound them. Neither paper runs the other's estimator, so which design is better in long horizons is open.
• Scope. COPC is the corpus's first run of an actor-side keep rule and a critic-side correction in one loop, the combination BRACE's future-work line asks about. BRACE ships with no actor-side rule. COPC's critic correction is not BRACE's, though, and COPC's longest responses are 16k tokens against BRACE's 8k–32k range.

Connections#

• Anchored Bellman-Residual Correction (BRACE): the sibling critic-side correction. Both make the same V^µ diagnosis, but BRACE builds a capped ratio product with a constant tail, while COPC uses per-token clipped weights with no product and adds an actor-side mask in the same loop.
• Asynchronous RL for LLMs: the domain hub. COPC is the first source to combine family 1 (a keep rule, DIS) with the fourth, critic-side family, and it adds an exact bias decomposition for the "controlled off-policy bias" DIS accepts.
• Single-Rollout Optimization: COPC's policy channel is SAO's DIS mask, used unchanged. Run as a rival-implemented baseline here, SAO collapses late on search, unlike its own paper's ~1000 stable steps.
• Group Relative Policy Optimization (GRPO): critic-free DAPO is COPC's group-relative baseline. It loses to the corrected critic by about 3–4 points but is one of only two baselines that never collapse, which is a data point for both sides of the critic pendulum.
• Staleness–Learning-Rate Scaling: COPC's uncorrected async PPO arm on Llama-3.2-3B at η = 10⁻⁶ trains 450 steps without collapse. Whether that conflicts with Song et al.'s S·η_max frontier depends on which staleness definition the comparison uses.
• The Stale Advantage Without a Critic: Does BRACE Have a Group-Relative Analogue?: with the critic removed, COPC's advantage collapses to one clipped terminal-token ratio while BRACE's capped product survives; also reads the DAPO-over-uncorrected-PPO margin as a critic-variance effect, since both uncorrected arms share the same µ-continuation bias at γ=λ=1

Open Questions#

• Does the coordination hypothesis hold outside the 3×3 grid and outside 3–4B dense models? The sweep spans only η ∈ {0.25, 0.5, 0.75}, on one benchmark and one backbone. A finer grid, or a run at 30B-A3B scale like SAO's, would show whether the strong-strong penalty and the anti-diagonal optimum are general, or an artifact of three points per axis.
• Which critic-side estimator wins in long horizons: COPC's per-token clipped weights with no product, or BRACE's k-capped product with a constant tail? BRACE predicts that per-token weights inject noise growing with the remaining horizon, and COPC's responses (≤16k tokens) may be too short for that noise to show. A head-to-head at 32k+ tokens, with the same actor mask, would settle it.
• Is the late-stage collapse COPC fixes caused by the critic, or by the actor-side corrections? In Figure 2, uncorrected async PPO and critic-free DAPO stay stable for 450 steps, while SAO, AReaL, V-trace and the actor-only ablation collapse. Rerunning the collapsers with a value-bias probe (BRACE's ĝ_π) would show whether the collapse tracks critic drift.

Sources#

• COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning: COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning. Authors: Zicheng Hu, Zhijian Zhou, Xuan Zhang, Yuchen Liu, Cheng Chen, Yuan Li, Qi Gu, Yan Feng, Hongyan Hao†, Chao Qu†, at Fudan University, Meituan and East China Normal University; the work was done during a Meituan internship. arXiv 2610.09597, 2026-10-07, 21pp, empirical.
• Sections cited: §3.2 (advantage staleness, Proposition 1), §3.3 (the two-channel setup), §4 (Propositions 2–3, Corollary 1, the coordination hypothesis), §5 (the COPC objective and critic loss), §6.1–6.4, Tables 1–3, Figures 1–2, and Appendix C.2.1 (the COPC, SAO, AReaL, V-trace and DAPO configurations).
• Figures 1 and 2 were viewed under the image two-pass rule. Figure 1's cell values are printed on the heatmap. All of Figure 2's step positions and score levels are approximate chart reads.
• Parse notes, in this page's convention. Tables 1–3 parsed clean and were reconciled cell for cell against pdftotext -layout (pages 7–8). Appendix Tables 4–5 show flattened superscripts, such as "1 × 10 - 5" meaning 1×10⁻⁵. verify.py flags these, but they are false positives, and only the actor LR (1×10⁻⁶) and max staleness (8/14) are cited from them. Stray layout tokens ("MLA", "FFN", "MoE") are welded into the body text at several points; they are figure-label residue and carry no content.