UlaCache provides unified logical addressing to scale sparse KV cache serving across GPU and CPU tiers

Decoupling sparse attention indexing from physical memory locations, UlaCache delivers up to 2.51x decoding throughput speedups on NVIDIA Blackwell by combining parallel bucketed LRU eviction with zero-overhead buffer retention.

Primary source: arXiv:2610.29820: https://arxiv.org/abs/2610.29820

What’s new: Serving long-context large language models under heavy user concurrency is severely bottlenecked by the memory footprint of the key-value (KV) cache. While dynamic sparse attention reduces arithmetic operations and memory bandwidth by reading only a subset of historical tokens per decoding step, the full KV cache must typically remain allocated in GPU memory, capping maximum concurrent batch sizes. In UlaCache: Unified Logical Addressing for Sparse LLM Serving (arXiv:2610.29820, 21 Oct 2026), researchers Isaac Zhu, Alan Cao, Hugh Yin, Rocky Wang, Orion Wang, Hscos Zhang, Jack Li, and Jason Zhao from Scitix.AI introduce UlaCache, a memory management runtime that provides a unified logical address space spanning GPU pools and host CPU memory.

UlaCache assigns stable logical token identifiers that decouple the attention layer’s token selection logic from physical hardware memory addresses. To minimize communication across PCIe and network interconnects, UlaCache maintains local GPU retention buffers that preserve recently attended KV entries across consecutive decoding steps, transferring only missing cache blocks on demand. When offloading to system RAM, UlaCache employs Parallel Bucketed LRU (PB-LRU), an asynchronous eviction policy that updates recency metadata and identifies replacement candidates in parallel across active requests. Tested on NVIDIA H200 and Blackwell B300 hardware configurations with speculative decoding enabled, UlaCache improves decoding throughput by 1.16x (data-parallel) and 1.70x (tensor-parallel) on H200, and by 1.73x (data-parallel) and 2.51x (tensor-parallel) on B300 compared to state-of-the-art serving baselines under identical memory budgets.

Why it matters: As frontier language models expand context windows to hundreds of thousands of tokens, high-density GPU high-bandwidth memory (HBM) becomes saturated by dormant cache entries. Existing offloading solutions either restrict GPU slots to mirrors of CPU storage or incur substantial re-indexing latency during dynamic token selection. By establishing stable logical indirection, UlaCache allows serving systems to leverage cost-effective system host RAM without sacrificing parallel decoding speed or sparse attention efficiency.

Caveats: The throughput gains reported depend heavily on access sparsity patterns and temporal locality in attention heads; dense attention sequences or workloads that randomly inspect early context tokens will experience increased PCIe transfer traffic. Furthermore, managing fine-grained logical-to-physical address mappings introduces metadata bookkeeping overhead that requires careful tuning under high token churn rates.