SparseDecoding: decoding-aware pruning for accurate and fast LLM generation

Wang et al. resolve distribution shifts in Hessian-guided LLM pruning by calibrating during autoregressive decoding, pairing it with an N:M sparse matrix-vector CUDA kernel.

Primary source: arXiv:2610.12327: https://arxiv.org/abs/2610.12327

What’s new: Large language model inference consists of two distinct computational regimes: prompt prefill, which is compute-bound and relies on sparse matrix-matrix (SpMM) multiplication, and autoregressive decoding, which is strictly memory-bandwidth bound and dominated by sparse matrix-vector (SpMV) operations. Existing training-free pruning methods use pre-collected natural sequences during prefill to compute Hessian or activation calibration matrices. However, during real decoding, the model processes its own autoregressively generated tokens. This discrepancy creates a distribution shift between calibration statistics and decoding activations, degrading generation quality. In SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference (arXiv:2610.12327, 8 Oct 2026), Wang et al. introduce SparseDecoding to align the pruning objective directly with decoding dynamics.

SparseDecoding constructs calibration matrices from layer-wise activations gathered during dense autoregressive generation, explicitly excluding the prefill phase. On the systems front, the authors develop a custom N:M semi-structured sparse matrix-vector kernel with bitmask indexing and fixed-step traversal, resolving the long-standing problem that tensor cores provide limited acceleration for SpMV workloads. Tested on Llama-3.1-8B, Llama-3.3-70B, and Qwen3 (14B and 32B), SparseDecoding consistently outperforms standard fixed-text calibration on long-form generation benchmarks while delivering up to 1.48x end-to-end wall-clock decoding speedup on NVIDIA A100 GPUs.

Why it matters: Serving large language models under long-context scenarios is heavily bottlenecked by high-bandwidth memory access during single-token generation. By identifying the calibration mismatch in existing pruning recipes and pairing decoding-aware calibration with a dedicated SpMV kernel, SparseDecoding translates theoretical parameter sparsity into tangible inference acceleration without sacrificing generation accuracy.

Caveats: Speedups require hardware support for semi-structured N:M sparsity and customized GPU kernel compilation. Furthermore, collecting calibration statistics across autoregressive rollouts increases the one-time offline calibration cost compared to standard single-pass prefill heuristics.