Two Clocks: why larger rollout batches learn slower but fail on schedule in RLVR

Square-root learning-rate scaling in multi-step GRPO preserves policy drift per sample rather than progress, causing larger batches to lose sample efficiency to curvature while collapsing at the same sample count.

Primary source: arXiv:2610.18853: https://arxiv.org/abs/2610.18853

What’s new: In reinforcement learning with verifiable rewards (RLVR), rollout generation dominates training cost, prompting practitioners to scale up rollout batch sizes using the square-root learning-rate rule ($\eta \propto \sqrt{B_{\mathrm{r}}}$) while taking multiple mini-batch optimizer steps per rollout. In Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards (arXiv:2610.18853, 13 Oct 2026), researchers from Carnegie Mellon University and the University of Toronto show that while square-root scaling preserves per-sample progress in single-step GRPO, multi-step training splits dynamics into two distinct clocks: a progress clock that slows down at large batches and a failure clock that stays locked to consumed samples.

Under a local quadratic loss model with noise-dominated Adam updates, the authors prove that square-root scaling holds first-order policy drift per sample constant, but larger batches cover that drift in fewer, longer optimizer steps that lose more progress to loss curvature. Unlike standard critical batch size theory, the resulting critical rollout batch $B^$ shrinks as the base learning rate increases: across controlled GRPO runs on Qwen2.5-Math-1.5B spanning a $16\times$ batch range (512 to 8,192 responses, $\mu = 8$ steps per rollout), raising the learning rate tenfold increases the sample-cost exponent from 0.15 to 0.44 and drops $B^$ from roughly 9,000 to 2,000 responses. Meanwhile, when PPO ratio clipping is disabled, training collapse is governed by accumulated policy drift rather than step count: a 4,096 batch collapses after 56 to 80 optimizer steps and a 512 batch after 648 steps, yet both fail after consuming 30,000 to 50,000 samples. Restoring ratio clipping trims the wider token importance-ratio spread of large batches (where 0.5% of clipped tokens carry over half the gradient norm), delaying collapse fourfold for large batches versus twofold for small batches without closing the curvature-induced progress gap. Guided by both clocks, a simple batch ramp that starts below $B^*$ (batch 512 for 24.6k samples) and switches to a large batch (4,096 or 8,192) before the predicted failure point matches the sample efficiency of the best fixed small batch while generating 75% of rollouts at high throughput.

Why it matters: Post-training reasoning models with GRPO requires balancing rollout generation throughput against sample efficiency and late-stage policy collapse. Separating curvature-limited progress from drift-triggered collapse explains why ratio clipping acts primarily as a stabilizer for large batches rather than the root cause of their slower learning, and gives infrastructure teams a principled batch-ramping schedule.

Caveats: Experiments are restricted to the Qwen2.5 model family (1.5B, 1.5B-Instruct, and 7B) on mathematical reasoning benchmarks (DAPO-Math-17k and DeepScaleR-Preview). The theoretical model relies on a local quadratic approximation whose Hessian curvature term is tested via scaling laws rather than direct spectral measurement. Finally, accounting is reported in consumed responses rather than wall-clock time or total token volume, which also depend on response length drift and distributed serving latency.