BitSparse: lossless GPU-resident activation compression cuts ReLU transformer VRAM by 31%

Fused Triton autograd kernels and a host-sync-free preallocated buffer compress sparse ReLU and ReLU^2 activation caches losslessly, saving nearly a third of per-token VRAM with under 1% overhead.

Primary source: arXiv:2610.22256: https://arxiv.org/abs/2610.22256

What’s new: Large language model training has increasingly gravitated toward sparse activation functions such as ReLU and squared ReLU (ReLU^2), which naturally induce 40% to 60% zero-valued activations across feed-forward network layers. However, modern training pipelines typically store intermediate activation tensors densely in high-bandwidth memory for gradient computation during the backward pass. Existing strategies to curb activation memory either incur steep compute penalties (activation checkpointing recomputes full forward passes, adding 30% to 33% runtime overhead) or introduce gradient noise (lossy activation quantization). In BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks (arXiv:2610.22256, 15 Oct 2026), University of Cambridge researchers Max Zhu, Bohang Sun, Pietro Liò, and Jialin Yu present BitSparse, a GPU-resident lossless compression format and execution framework tailored for sparse neural activations.

BitSparse compresses intermediate activation caches directly on GPU without host-device synchronization. The system implements custom Triton kernels fused into autograd operations, packing non-zero values and sparse index bitmasks in registers during forward execution and reconstructing exact original values during backward passes. The design supports both BF16 and FP8 activation formats, incorporating an optional sign-bit packing mechanism that approaches theoretical Shannon entropy limits. To resolve the memory fragmentation and CPU-GPU synchronization bottlenecks typically caused by dynamic variable-sized compressed tensors, BitSparse introduces a preallocated GPU buffer pool that eliminates host synchronization entirely. On 16,000 x 21,504 intermediate projections, BitSparse reduced backward peak activation memory from 1,312.5 MiB to 698.25 MiB. Across full model training runs on Modded-NanoGPT and Nemotron-H architectures, BitSparse lowered per-token VRAM requirements by 31% and 30% respectively, while adding less than 1% runtime overhead at standard sequence lengths.

Why it matters: Training memory constraints directly dictate achievable batch sizes and context lengths on modern accelerator clusters. Because BitSparse is mathematically lossless, it introduces zero degradation to optimization dynamics, model convergence, or downstream perplexity, unlike lossy 8-bit or 4-bit activation quantization schemes. By delivering substantial memory savings with negligible execution overhead, BitSparse provides a drop-in systems optimization for frontier architectures adopting sparse activation functions.

Caveats: BitSparse is specifically engineered for activation functions that produce structural sparsity, such as ReLU, ReLU^2, and top-k routing in mixture-of-experts models; it yields no benefit for dense activations like GELU or SwiGLU without prior thresholding. In addition, memory savings scale with activation sparsity: models operating at low sparsity ratios (below 30%) will see diminishing memory reductions relative to the metadata indexing overhead. Finally, the implementation relies on Triton kernels optimized for modern NVIDIA architectures (Ampere and Hopper), and performance across AMD or custom ASIC accelerators remains uncharacterized.