KronLSTM: Kronecker delta rule stabilizes long-context associative memory in linear attention

Extending xLSTM with rank-1 Kronecker delta updates produces an associative memory that provably converges as a steady-state Markov chain with uniformly bounded norm, outperforming DeltaNet on long sequences.

Primary source: arXiv:2610.26834: https://arxiv.org/abs/2610.26834

What’s new: Linear attention architectures and recurrent state models offer linear computational complexity in sequence length, making them compelling alternatives to standard softmax self-attention for long-context language modeling. However, maintaining stable associative matrix memories across tens of thousands of tokens remains challenging: standard delta-rule updates often suffer from destructive interference or numerical instability, requiring complex normalization heuristics to prevent state explosion. In Kronecker Delta Rule for Long Context Linear Attention (arXiv:2610.26834, 19 Oct 2026), researchers Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer, Sebastian Böck, and Sepp Hochreiter introduce KronLSTM, an associative memory linear attention model governed by the Kronecker Delta Rule.

KronLSTM extends the extended Long Short-Term Memory (xLSTM) framework by formulating a minimally invasive memory-revision mechanism. Rather than applying unconstrained matrix overwrites, the Kronecker Delta Rule updates memory associations by altering matrix state exclusively along a single rank-1 direction in matrix space. This targeted modification preserves previously learned associations while correcting specific key-value mappings. The authors provide a theoretical proof establishing that the resulting recurrent memory dynamics can be modeled as a Markov chain that converges to a well-behaved steady-state distribution. Crucially, the expected Frobenius norm of the memory matrix remains uniformly bounded across arbitrarily long sequence lengths without requiring layer normalization, weight clamping, or running RMS normalizers. In empirical evaluations against leading linear recurrent architectures, including Gated DeltaNet and standard xLSTM variants, KronLSTM achieved superior language modeling perplexity and synthetic associative recall accuracy, demonstrating pronounced performance advantages on extended sequence lengths.

Why it matters: Developing sub-quadratic recurrent models that match softmax transformer quality without succumbing to memory saturation or numerical divergence is a central pursuit in modern language modeling. By establishing an analytically tractable update rule with proven norm bounds and minimal interference, KronLSTM provides a mathematically grounded memory architecture for efficient continuous long-sequence processing.

Caveats: While recurrent inference requires constant memory per step during generation, associative matrix updates involve outer products that are more memory-bandwidth intensive during parallel training than simple scalar-decay state models like Mamba. Additionally, although the steady-state Markov bound prevents exploding memory norms, finite matrix capacity still imposes a theoretical ceiling on the total number of distinct key-value pairs retrievable without error on dense retrieval tasks.