BeNice unlocks zero-overhead KV cache sharing across recurrent depth in looped transformers

Combining stochastic depth pretraining with the BeNice post-training broadcast enables looped transformers to share KV caches across recurrences, boosting throughput by 24% and cutting p90 TTFT by 84%.

Primary source: arXiv:2610.31633: https://arxiv.org/abs/2610.31633

What’s new: Looped transformer architectures offer a compelling parameter-efficient paradigm: they expand computational depth and expressive capacity by recurrently passing representations through an identical block of layers. However, looped models suffer from an inference-memory gap: each recurrence creates its own independent key-value (KV) cache, multiplying memory requirements linearly with loop depth and negating memory savings during serving. In BeNice! Closing Looping’s Inference-Memory Gap via KV-Sharing (arXiv:2610.31633, 22 Oct 2026), researchers Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick, and Daniel Y. Fu from UC San Diego and Stanford introduce a two-stage method to enable depth-wise KV cache sharing without degradation in model quality.

While prior research noted that final-recurrence KV caches could occasionally be reused across earlier recurrences, this property, termed KV-shareability, was fragile and failed to emerge reliably even across identical architectures. The authors resolve this through a two-stage training scheme: first, stochastic depth during pre-training induces contractive residual dynamics, pulling hidden representations across recurrences toward a shared manifold. Second, BeNice!, a post-training procedure, broadcasts the final recurrence’s KV cache into the initial recurrence, training the initial layers to generate compatible residual representations. In long-context serving evaluations, BeNice! matches the perplexity of unshared fine-tuning while enabling complete KV cache reuse across depth. It composes seamlessly with Multi-Head Latent Attention (MLA), Kimi delta attention, and 4-bit KV quantization (NVFP4, MXFP8), boosting cache hit rates by up to 31.4%, increasing serving throughput by 23.6%, and slashing p90 time-to-first-token (TTFT) by 83.6%.

Why it matters: As reasoning models scale test-time compute through iterative looping and recurrent reasoning, inference memory footprint becomes the primary deployment bottleneck. BeNice! demonstrates that recurrent parameter efficiency does not have to come at the expense of runtime memory expansion. By enabling full KV cache sharing across recurrent steps, the method brings looped architectures into practical contention with standard feedforward models for memory-constrained production inference.

Caveats: The method requires incorporating stochastic depth during pre-training, meaning existing off-the-shelf looped checkpoints cannot be adapted solely through the post-training step without risk of perplexity regression. Additionally, latency improvements depend heavily on KV cache reuse and memory-bound serving conditions; compute-bound batch regimes may experience smaller throughput gains.