Bitware News
History · Ch. 092026-09-21

Sequences Before Attention

Before transformers, language and other sequences became trainable transduction problems through simple recurrent nets, LSTM, encoder–decoder seq2seq, and Bahdanau’s soft alignment—attention as an RNN add-on, not a 2017 invention.

History

Primary source: Hochreiter & Schmidhuber, LSTM (1997): https://doi.org/10.1162/neco.1997.9.8.1735

Earlier chapters treated multilayer supervised learning as a shared gradient tool and traced how first-order training methods accumulated from steepest descent to Adam. Those accounts still left open a different architectural question: how to make variable-length sequences—language, speech, and other ordered signals—into differentiable problems whose length need not be fixed in advance. The path that answered that question did not begin with the 2017 transformer. It ran through simple recurrent nets that carried context in time, through LSTM’s gated memory cells aimed at vanishing gradients, through encoder–decoder sequence-to-sequence models that treated translation as end-to-end transduction, and through Bahdanau, Cho, and Bengio’s soft-alignment attention as an add-on to recurrent encoder–decoders. This chapter follows that line. Full self-attention without recurrence belongs to a later chapter; so do ImageNet-scale vision and the later language-model stacks.

Elman 1990: structure in time, not in a fixed window

In 1990, Jeffrey L. Elman published Finding Structure in Time in Cognitive Science (vol. 14, no. 2, pp. 179–211). The problem he states is representational: many human behaviors are temporal, yet connectionist models often treated time by converting it into a spatial pattern—a fixed window of past inputs laid out as a larger static vector. Elman’s alternative is to represent time implicitly, by its effects on processing. Hidden-unit activations are fed back into the network as context units on the next step, so the internal state at time (t) depends on both the current input and the prior hidden pattern. The resulting architecture is what later literature calls a simple recurrent network (SRN), or Elman net: context units copy the previous hidden layer and provide a dynamic memory that is updated, not hand-segmented into a sliding window.

Elman reports simulations that range from a temporal exclusive-or task to word-prediction problems in which the network discovers syntactic and semantic structure from sequential exposure. The learned hidden representations are highly context-dependent yet still support generalizations across classes of items—lexical categories and type/token distinctions among them. Memory, in this framing, is not a separate buffer bolted onto a feedforward net; it is bound up with task processing.

Two careful historical notes follow from the paper itself. First, Elman explicitly develops a proposal “first described by Jordan (1986)”—Michael I. Jordan’s ICS technical report Serial Order: A Parallel Distributed Processing Approach (ICS Report 8604, May 1986)—which used recurrent links for serial action. Claiming that Elman nets were “the first recurrent nets” is folklore; recurrence as a modeling idea is older, and Elman’s own citation trail points to Jordan among proximate connectionist precedents. Second, Elman nets showed that gradient-trained recurrence could discover temporal structure. They did not, by themselves, solve the long-lag credit-assignment problem that dominates the next section.

Vanishing gradients and LSTM: constant error carousels

Training recurrent nets by backpropagation through time (or closely related real-time recurrent learning) exposes a pathology that is easy to state and hard to live with. Error signals that must travel many time steps tend either to shrink exponentially toward zero—vanishing gradients—or to grow without bound—exploding gradients. In the vanishing case, learning long-range dependencies becomes impractically slow or fails; the network cannot assign credit across the lag even when the architecture could in principle store the needed information.

A clean early analysis of that difficulty is Sepp Hochreiter’s June 1991 Diplomarbeit at the Technische Universität München, Untersuchungen zu dynamischen neuronalen Netzen (supervisor Jürgen Schmidhuber). Written in German, the thesis examines error backflow in recurrent nets and motivates architectures that keep error flow from decaying over long delays. The LSTM paper later cites this analysis directly; an English survey treatment appears in Hochreiter, Bengio, Frasconi, and Schmidhuber’s chapter Gradient Flow in Recurrent Nets (in Kremer and Kolen’s A Field Guide to Dynamical Recurrent Neural Networks, IEEE Press, 2001). The verified claim for 1991 is the diploma analysis of vanishing (and exploding) error backflow—not a claim that the full LSTM cell already existed in that thesis under its later name.

The architectural answer that became standard is Sepp Hochreiter and Jürgen Schmidhuber’s Long Short-Term Memory, published in Neural Computation in November 1997 (vol. 9, no. 8, pp. 1735–1780). LSTM introduces memory cells whose internal recurrence is designed to enforce a constant error carousel: a path along which error can flow across many time steps without the multiplicative decay that kills ordinary recurrent units. Multiplicative gate units learn to open and close access to that path—controlling what is written, read, and (in later refinements) forgotten. The paper reports that LSTM can bridge minimal time lags in excess of a thousand discrete steps on designed tasks, compares favorably to BPTT, RTRL, Elman nets, and other contemporary recurrent methods on artificial benchmarks, and remains local in space and time with (O(1)) cost per time step and weight.

What LSTM did not do is erase recurrent nets overnight or abolish vanishing gradients as a phenomenon in every architecture. It made long-lag sequence learning tractable for a large class of problems by changing the cell’s error-flow geometry. Later practice still needed careful initialization, clipping for explosions, gated variants, and—eventually—attention and non-recurrent alternatives. The folklore that “LSTM solved vanishing gradients completely and made RNNs obsolete” collapses a major engineering advance into a finished story.

Encoder–decoder seq2seq: transduction as a trained map

By the mid-2010s, the research question shifted from “can a recurrent net remember?” to “can we map one sequence to another of different length end-to-end?” Machine translation is the flagship instance: a source sentence of length (T) must become a target sentence of length (T’), with no guarantee that (T = T’), and with word order and morphology that resist fixed templates.

An important 2014 statement of the encoder–decoder idea is Kyunghyun Cho, Bart van Merriënboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio’s Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (EMNLP 2014, pp. 1724–1734; arXiv:1406.1078). An RNN encoder reads a variable-length input into a continuous vector; an RNN decoder generates a variable-length output conditioned on that vector. The paper also introduces a gated hidden unit—later widely called the GRU—that combines update and reset gates in a lighter cell than classical LSTM. In their experiments the encoder–decoder improves phrase-based statistical MT when used to score phrase pairs and to provide continuous phrase representations. The architecture is already clearly a trained sequence transducer; it is not yet the full attention model of the next section.

The pure neural sequence-to-sequence result that made the framing travel in the NIPS/NeurIPS community is Ilya Sutskever, Oriol Vinyals, and Quoc V. Le’s Sequence to Sequence Learning with Neural Networks (NIPS 2014; Advances in Neural Information Processing Systems 27, pp. 3104–3112; arXiv:1409.3215). A deep LSTM encodes the source into a fixed-dimensional vector; another deep LSTM decodes the target from that vector. On WMT’14 English–French, they report a BLEU score of 34.8 on the full test set (penalized for out-of-vocabulary words), against 33.3 for a phrase-based SMT baseline; LSTM reranking of SMT’s 1000-best list reaches 36.5. A striking practical detail: reversing the source word order (not the target) markedly improved training, by creating more short-term source–target dependencies and easing optimization. The paper treats sequence transduction as a general neural problem with minimal assumptions on structure—exactly the move that later architectures inherit even when they discard recurrence.

The shared bottleneck in both the Cho-style encoder–decoder and Sutskever-style seq2seq is the fixed-length vector that must summarize the entire source before decoding begins. That compression is efficient when sentences are short and the summary is rich; it becomes a suspected failure mode as sources lengthen. Soft alignment is the response to that suspicion.

Bahdanau attention: soft alignment over encoder states

In September 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio posted Neural Machine Translation by Jointly Learning to Align and Translate (arXiv:1409.0473); the work was presented as an oral paper at ICLR 2015. Their conjecture is architectural: forcing the encoder to pack the whole source into one fixed-length vector is a bottleneck. They extend the encoder–decoder by letting the decoder, at each target step, compute a soft search over source positions—a weighted combination of encoder annotations (bidirectional RNN states in their design)—without committing to hard segment boundaries in advance.

Mechanically, that soft search is what this series means by attention in the 2014/2015 sense. An alignment model scores how well each source position matches the decoder’s current state; a softmax turns those scores into weights; the weights form a context vector that conditions the next target word. Alignment and translation are trained jointly. On English–French translation they report performance comparable to strong phrase-based systems, and qualitative inspection shows soft alignments that match linguistic intuition (for example, attending to the source words that license the word being generated).

Three distinctions matter for later chapters. First, this attention is an add-on to recurrent encoder–decoders, not a replacement for recurrence. Second, it is soft alignment over encoder states—content-based addressing of the source—not the full multi-head self-attention stack of Vaswani et al. 2017. Third, therefore, the slogan “transformers invented attention” is folklore. Transformers generalized and scaled a family of attention mechanisms and removed recurrence as the backbone; they did not introduce the idea of differentiable soft alignment for neural MT. Bahdanau et al. already made sequence transduction a trained encoder–decoder problem in which the decoder could look back selectively at the source.

Caveat

Three retellings especially distort this material.

First, “transformers invented attention.” Soft alignment over encoder states appears in Bahdanau, Cho, and Bengio’s ICLR 2015 (arXiv 2014) neural MT work as an extension of RNN encoder–decoders. The 2017 transformer paper is a later architectural reorganization—self-attention without recurrence—not the origin of attention as a concept in neural sequence models.

Second, “LSTM solved vanishing gradients completely and made RNNs obsolete overnight.” Hochreiter and Schmidhuber 1997 redesigned error flow through gated constant-error-carousel cells and demonstrated long-lag learning that prior recurrent methods struggled with. Vanishing and exploding gradients remained live issues for deep and recurrent nets; gated RNNs, careful training practice, and later non-recurrent models are continuations of that struggle, not proof that 1997 ended it.

Third, “Elman nets were the first recurrent nets.” Elman 1990 is a foundational, widely taught statement of simple recurrent nets with context units for discovering structure in time. Elman himself situates the proposal relative to Jordan 1986, and recurrent architectures more broadly have an earlier paper trail. The historical point of Elman for this chapter is not absolute priority; it is that gradient-trained recurrence with a dynamic context state became a workable cognitive and engineering tool for sequential structure before LSTM, seq2seq, and attention completed the transduction stack.

Sources

Primary

Historical / technical reference