ReadKV implements query-adaptive quantization to cut KV cache memory bandwidth

Bhandari et al. introduce ReadKV, storing KV caches in progressive codes to fetch precision selectively based on query attention, cutting read bandwidth by half with minimal perplexity impact.

Primary source: arXiv:2610.11245: https://arxiv.org/abs/2610.11245

What’s new: In long-context large language model serving, memory bandwidth consumption from reading key-value (KV) caches during autoregressive decoding dominates inference latency. Traditional KV cache compression methods apply uniform low-bit quantization or static token eviction, which either degrade fine-grained reasoning or permanently discard historical information before future queries arrive. In Read What Matters: Query-Adaptive Quantization for KV Caches (arXiv:2610.11245, 8 Oct 2026), Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, and Shiva Kasiviswanathan present ReadKV, a query-adaptive cache framework that decouples retained storage capacity from per-query read payloads.

ReadKV stores key and value states in progressive codes where bit prefixes support variable reconstruction precisions without modifying stored representations. During generation, each incoming query dynamically allocates key-channel prefix budgets to estimate attention weights; the resulting attention distribution then allocates value-token prefix budgets so that only high-attention tokens are fetched at higher bit-depths. The authors formulate this allocation as a calibrated distortion optimization problem under fixed read budgets and prove that query-dependent readers strictly outperform all query-independent codecs. Across six base language models, reading an average of 4 bits from an 8-bit cache increases C4 perplexity by at most 0.66% while utilizing approximately one quarter of the logical reads of a 16-bit cache. On an NVIDIA A10G GPU running an 8K-token workload, ReadKV achieves a 39% latency reduction compared to TurboQuant.

Why it matters: For long-context serving and multi-turn agent conversations, transferring KV cache tensors between high-bandwidth GPU memory and compute units is the primary performance bottleneck. By enabling queries to fetch high precision only for the specific tokens and channels they attend to, ReadKV achieves the fidelity of high-bit retention alongside the serving speed and bandwidth efficiency of ultra-low-bit quantization.

Caveats: ReadKV requires custom GPU kernel implementations to efficiently gather variable-length progressive code prefixes without introducing memory coalescing penalties. Furthermore, on workloads with very short context lengths or large decoding batch sizes where model weight bandwidth dominates cache access, the latency benefits of progressive reading are less pronounced.