Bitware News
History · Ch. 122026-09-24

Transformers and Scaling

The 2017 transformer reorganized sequence modeling around self-attention without recurrence; GPT and BERT are two objectives on that backbone; Kaplan and Chinchilla scaling laws are empirical regularities about compute, data, and parameters—not a slogan that bigger is always better.

HistoryInfraEval

Primary source: Vaswani et al., NIPS 2017 (Transformer): https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

Earlier chapters left sequence modeling with recurrent encoder–decoders and soft alignment (Chapter 9) and left representation learning with a pretrain/finetune recipe whose NLP half already split into autoregressive and masked objectives (Chapter 11). This chapter closes the planned serial at the architecture and evaluation regime that still dominate large language-model practice: the transformer—self-attention without recurrence as the backbone—and the scaling laws that treat cross-entropy (and related metrics) as predictable functions of parameters, data, and compute. The argument is methodological, not metaphysical. The transformer reorganized how sequences are computed; GPT- and BERT-style models are two uses of that backbone; Kaplan et al. 2020 and Hoffmann et al. 2022 (Chinchilla) are empirical fits and corrections about resource allocation. None of them completes a science of intelligence. They mark where the present stack sits.

Attention Is All You Need: self-attention without recurrence

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin’s Attention Is All You Need (NIPS 2017; arXiv:1706.03762) proposes a sequence transduction architecture—the Transformer—based solely on attention mechanisms, “dispensing with recurrence and convolutions entirely.” The overall shape remains encoder–decoder: an encoder maps a source sequence to continuous representations; a decoder generates the target autoregressively. What changes is the computational substrate. Both stacks are built from multi-head self-attention and position-wise feed-forward layers, with residual connections and layer normalization, rather than from recurrent steps that process tokens one after another.

Three mechanisms carry the architectural load. Scaled dot-product attention computes compatibility between queries and keys, scales by the square root of the key dimension, softmax-normalizes, and forms a weighted sum of values. Multi-head attention runs several such attentions in parallel with different projections and concatenates the results, so the model can jointly attend to information from different representation subspaces. Positional encodings—sinusoidal functions of token position in the published design, with a learned-embedding alternative found nearly as good—are added to embeddings because the model has no recurrence or convolution with which to recover order. The decoder’s self-attention is masked so that positions cannot attend to future tokens, preserving autoregression; encoder–decoder attention lets every decoder position attend over the encoder outputs, in the lineage of soft alignment rather than as a reinvention of it.

The empirical claims are translation results under fixed training budgets. On WMT 2014 English–German, their big model reports 28.4 BLEU, improving over prior published results including ensembles; on WMT 2014 English–French, 41.0 BLEU as a single-model result after about 3.5 days on eight NVIDIA P100 GPUs. Parallelism is part of the engineering point: without sequential recurrence along the length dimension, training can exploit hardware that favors batched matrix multiplies. That is an architectural and systems claim, not a claim that recurrent nets ceased to exist the day the paper appeared.

Relation to Chapter 9 must be kept sharp. Bahdanau, Cho, and Bengio’s soft alignment (ICLR 2015; arXiv 2014) already made differentiable attention an add-on to recurrent encoder–decoders. The 2017 transformer generalizes attention into the backbone—multi-head self-attention within each stack, plus cross-attention—and removes recurrence as the primary sequential engine. The slogan “attention was invented in 2017” is folklore. What 2017 contributed, carefully stated, is a full self-attentional encoder–decoder that made non-recurrent sequence transduction the default research path for a large class of problems.

Shared backbone, two objectives: GPT and BERT

Chapter 11 already introduced the complementary NLP pretraining tracks. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever’s Improving Language Understanding by Generative Pre-Training (OpenAI, 2018) trains a Transformer with a left-to-right language-modeling objective and then finetunes. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova’s BERT (NAACL-HLT 2019; arXiv:1810.04805) pretrains a bidirectional Transformer encoder with masked language modeling (and next-sentence prediction) before finetuning. The historical point for this chapter is architectural, not a second pass over the pretrain/finetune recipe: both lines reuse the transformer stack as a shared computational backbone and differ primarily in objective and attention masking—causal autoregression versus bidirectional context under masks. Encoder-only, decoder-only, and encoder–decoder variants proliferate later; the GPT/BERT split is already enough to see that “the transformer” is not one training recipe.

Kaplan et al. 2020: power laws in N, D, and C

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei’s Scaling Laws for Neural Language Models (arXiv:2001.08361, 23 January 2020) studies how autoregressive Transformer language-model cross-entropy loss scales with non-embedding parameter count (N), dataset size (D) (tokens), and training compute (C). The central empirical claim is that loss follows smooth power laws in each factor when that factor is the limiting bottleneck, with trends reported over many orders of magnitude. In their summary fits, roughly (L(N) \propto N^{-\alpha_N}) with (\alpha_N \approx 0.076), (L(D) \propto D^{-\alpha_D}) with (\alpha_D \approx 0.095), and (L(C_{\min}) \propto C_{\min}^{-\alpha_C}) with (\alpha_C \approx 0.050), when the other resources are not the binding constraint. Architectural shape (width versus depth) matters little within a wide band once total non-embedding size is fixed.

They also fit how loss depends jointly on (N) and (D), and how a fixed compute budget should be allocated. Under their analysis, optimally compute-efficient training puts most new compute into larger models, trains on a relatively modest amount of data, and stops well short of convergence: empirically they report roughly (N \propto C^{0.73}) and data requirements growing much more slowly ((D \sim C^{0.27}) in the compute-efficient sketch). Larger models appear more sample-efficient. These are fitted regularities for loss under their training setup and tokenization—not a proof that “bigger is always better” on every downstream metric, nor a guarantee that the same exponents travel unchanged to every architecture, data mixture, or evaluation protocol.

Hoffmann et al. 2022 (Chinchilla): the data correction

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and coauthors’ Training Compute-Optimal Large Language Models (arXiv:2203.15556, 29 March 2022)—widely called the Chinchilla paper—reopens the allocation question. Training over 400 models from roughly 70 million to over 16 billion parameters on 5 to 500 billion tokens, they conclude that for compute-optimal training, model size and training tokens should be scaled roughly equally: for every doubling of parameters, tokens should roughly double. Relative to Kaplan-style extrapolations that grew parameters much faster than data, they argue that many contemporary large models were undertrained on data for their compute budgets. Their projected frontier (across three estimation approaches) clusters near equal exponents for parameters and tokens with compute—on the order of (N \propto C^{0.5}), (D \propto C^{0.5})—rather than Kaplan’s steeper preference for parameters.

The test case is Chinchilla: about 70 billion parameters trained on 1.4 trillion tokens, matched to the compute budget of DeepMind’s Gopher (280 billion parameters, ~300 billion tokens). Chinchilla outperforms Gopher and several larger published models on a suite of downstream evaluations; the abstract highlights about 67.5% average accuracy on MMLU, a substantial gain over Gopher despite far fewer parameters. The methodological moral is allocation and evaluation, not abolition of scale. Chinchilla did not “kill scaling.” It corrected a compute–data–parameter tradeoff and showed that a smaller, more thoroughly trained model can beat a larger, undertrained one at the same FLOP budget. Later practice also explores deliberate “overtraining” (more tokens than a Chinchilla-optimal point) for inference-cost reasons—outside this chapter’s scope, but enough to underline that scaling laws are engineering curves, not commandments.

On infrastructure, a single checkable note: Vaswani et al. report training on eight P100 GPUs; Hoffmann et al. state that the models in their analysis were trained on TPUv3/TPUv4 clusters (citing Jouppi et al.’s TPU line). Scaling laws presuppose that such accelerators and interconnects exist. They do not require invented FLOP folklore beyond what the papers themselves report.

Where the serial stops

Put as one closing claim: the present methodological horizon for large language models is a transformer-family backbone, pretrained at scale under autoregressive or related objectives, evaluated and planned with empirical scaling relationships among compute, data, and parameters—always subject to revision when the fits change. This serial is a history of methods, not a completed science of intelligence. Catalogs of every post-2022 model, full treatments of preference tuning and RLHF, and multimodal generative stacks belong to other accounts. The planned chapters end here, at the architecture and the scaling-evaluation regime that still organize the field’s default stack—with caveats, not a victory lap.

Caveat

Four retellings especially distort this material.

First, “attention was invented in 2017.” Soft alignment over encoder states is already in Bahdanau, Cho, and Bengio’s neural MT work (Chapter 9). Vaswani et al. replace recurrence with multi-head self-attention as the backbone and scale the idea; they do not originate differentiable attention.

Second, “scaling laws prove bigger is always better” / “Chinchilla killed scaling.” Kaplan et al. fit power laws for loss versus (N), (D), and (C) and recommended allocating most new compute to parameters under their assumptions. Hoffmann et al. revise the optimal ratio of tokens to parameters and show undertrained large models can lose to smaller, better-trained ones at equal compute. Neither result is a metaphysical slogan; both are empirical allocation claims under stated metrics and setups.

Third, “transformers made RNNs vanish overnight.” The 2017 paper demonstrated superior translation quality with greater parallelizability under their training regime. Recurrent and convolutional sequence models continued in research and production for years; displacement was gradual and domain-dependent, not instantaneous extinction.

Fourth, treating this serial as a finished theory of intelligence. The chapters trace methods—optimization, kernels, deep nets, sequences, pretraining, transformers, scaling evaluation. They do not close the scientific question of what intelligence is or guarantee that the present stack is the end of the line.

Sources

Primary

Hardware / infra (brief)