DLite: history-free speculative decoding boosts agentic LLM inference throughput by up to 70%

By reusing target model hidden states and eliminating the draft model's expanding historical KV cache, DLite overcomes speculative decoding degradation in multi-turn agentic environments.

Primary source: arXiv:2610.23787: https://arxiv.org/abs/2610.23787

What’s new: Speculative decoding accelerates large language model generation by drafting candidate tokens with a lightweight assistant model and verifying them in parallel with a target model. While effective for single-turn generation, speculative decoding degrades severely in agentic workloads that involve rapidly expanding context windows and prolonged multi-turn interaction loops. In these environments, draft models experience up to an 80% drop in token acceptance length, while storing and updating the draft model’s historical Key-Value (KV) cache creates significant memory and memory-bandwidth overhead. In DLite: History-free Speculative Decoding for Agentic AI (arXiv:2610.23787, 16 Oct 2026), researchers Jiaming Xu, Hanzhen Wang, Tianlang Zhao, and colleagues from Tsinghua University and Infinigence AI introduce DLite, an architecture that eliminates the draft model’s historical KV cache entirely.

DLite circumvents history maintenance by directly reusing contextual hidden representations from the target model. Instead of maintaining long historical token sequences across turns, the draft module formulates candidate generation as a local prediction task conditioned on the target model’s most recent representations. To maintain draft accuracy across multi-token speculation blocks, DLite adds a lightweight sequential head that restores intra-block causal dependencies at negligible compute cost, optimized using a tailored two-stage training scheme. Evaluated across general, long-context, and multi-turn agentic workloads, DLite maintains an average acceptance length of 2 to 6 tokens and delivers up to a 70% throughput improvement over prior state-of-the-art speculative decoders such as DSpark.

Why it matters: Multi-turn autonomous agents frequently cycle between environment feedback, scratchpad updates, and tool call responses, causing KV caches to swell and straining GPU memory capacity. DLite drastically reduces speculative memory traffic by eliminating redundant draft cache storage, making fast speculative decoding practical for memory-constrained agent serving infrastructure.

Caveats: Reusing target model hidden states couples the drafting module closely to the specific target model layer dimensions and representations, requiring dedicated draft head training for each target model architecture. Furthermore, the local prediction head may see diminishing acceptance lengths on tasks requiring tight, complex syntactic constraints across distant tokens not captured in recent local state transitions.