dSpark accelerates diffusion language models by distilling uncertainty-resolving unmasking schedules

Myrzakhan and Shen introduce dSpark, a schedule distillation framework for diffusion LLMs that prioritizes tokens resolving global sequence uncertainty, achieving up to 10.12x speedups on reasoning benchmarks.

Primary source: arXiv:2610.35804: https://arxiv.org/abs/2610.35804

What’s new: Diffusion large language models (dLLMs) offer promising alternatives to standard autoregressive decoders by enabling parallel token generation, controllable infilling, and bidirectional revision. However, their practical deployment has been bottlenecked by slow iterative sampling schedules that require dozens or hundreds of sequential denoising rounds to produce coherent output. Conventional sampling heuristics prioritize revealing tokens with the highest individual prediction confidence, yet high-confidence tokens often convey little new contextual information to the remaining masked positions. In Learning When to Commit: Schedule Distillation for Fast Diffusion Language Models (arXiv:2610.35804, 26 Oct 2026), Aidar Myrzakhan and Zhiqiang Shen introduce dSpark, an unmasking schedule distillation framework designed to accelerate diffusion language generation.

The core insight behind dSpark is that an optimal unmasking schedule should commit to pivotal tokens whose revelation substantially reduces predictive entropy across the remaining ungenerated positions, even if those pivotal tokens initially exhibit lower marginal certainty. To operationalize this, the authors construct information-aware unmasking trajectories that measure how committing to candidate tokens accelerates the predictability of neighboring tokens. They then distill these high-influence schedules directly into the diffusion model using a specialized distillation objective combined with a Kullback-Leibler (KL) divergence regularizer to maintain output quality and sample diversity. Evaluated on challenging mathematical reasoning (GSM8K) and code generation tasks, dSpark achieves a 10.12x inference speedup on LLaDA-8B-Instruct and a 5.12x speedup on Dream-7B-Instruct over standard sampling baselines, all while matching or exceeding original task accuracy.

Why it matters: Serving costs and time-to-first-token latency are the defining operational hurdles for deploying non-autoregressive and diffusion foundation models in production environments. By learning an efficient, distilled unmasking order that maximizes mutual information gain per step, dSpark dramatically shrinks the number of sampling steps required for convergence. This brings the serving latency of diffusion language models into direct competition with optimized autoregressive engines, while preserving the structural advantages of bidirectional generation.

Caveats: The schedule distillation process requires generating offline trajectory data and conducting an auxiliary distillation training phase, which adds upfront training computation. Additionally, while the method demonstrates dramatic acceleration on structured reasoning and coding tasks where token interdependencies are strong, the speedup gains may be more modest in open-ended creative text generation where uncertainty is broadly distributed across the vocabulary.