Bitware News
History · Ch. 112026-09-23

Pretrain, Finetune, Self-Supervision

Supervision is scarce: the modern stack learns reusable representations from large unlabeled or weakly labeled data—layerwise pretraining, ImageNet transfer, word2vec, BERT-style masked LMs, and contrastive vision—then adapts them, without collapsing the lineage into a BERT founding myth.

History

Primary source: Devlin et al., NAACL 2019 (BERT): https://aclanthology.org/N19-1423/

Earlier chapters treated the 2012 ImageNet ConvNet win as a supervised scale-and-benchmark event and left open a different economic and methodological question: once deep nets work, where do the labels come from? Full supervision is expensive. The stack that became default after the mid-2010s treats large unlabeled (or weakly labeled) corpora as a way to learn reusable representations, then adapts those representations to a target task with far fewer labeled examples. That recipe has several historically distinct strands—layerwise unsupervised pretraining of deep nets, ImageNet features reused off-the-shelf, distributional word vectors, masked and autoregressive language-model pretraining, and contrastive self-supervision in vision. Collapsing all of them into “BERT invented pretraining” is folklore. This chapter traces the lineage without re-litigating the deep-learning turn or the transformer architecture as such.

Back-pointer: layerwise unsupervised pretraining

Chapter 10 already recorded Geoffrey E. Hinton and Ruslan R. Salakhutdinov’s Reducing the Dimensionality of Data with Neural Networks (Science, vol. 313, no. 5786, 28 July 2006, pp. 504–507). A deep autoencoder was pretrained greedily as a stack of restricted Boltzmann machines and then fine-tuned, yielding low-dimensional codes that preserved structure better than principal components analysis on several datasets. That paper is a landmark for deep belief nets and greedy layerwise training as an answer to hard-to-train deep supervised nets. It is not the modern “pretrain on a web-scale corpus, then finetune a transformer on GLUE”: the data regimes, objectives, and architectures differ. The historical point for this chapter is narrower. By the mid-2000s, unsupervised pretraining followed by supervised adaptation was already a named research strategy for deep networks. Later transfer learning did not invent the idea of learning representations before the scarce labeled task; it changed the data, the objectives, and the reuse pattern.

ImageNet features as transferable representations

After AlexNet-scale supervised training on ImageNet, a different transfer story emerged in vision: treat a large labeled recognition net as a feature extractor for tasks that were never its training objective. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell’s DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition (ICML 2014; Proceedings of Machine Learning Research 32, pp. 647–655) evaluates activations from a deep ConvNet trained fully supervised on a fixed object-recognition set, reused for novel tasks—scene recognition, domain adaptation, fine-grained recognition—where there may be too little labeled (or unlabeled) data to train a deep net from scratch. They compare which network levels define a useful fixed feature and release DeCAF as open-source activation features and network parameters. The emphasis is re-purposing fixed deep activations, not inventing end-to-end finetuning as the only mode of transfer.

A closely related demonstration is Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson’s CNN Features Off-the-Shelf: An Astounding Baseline for Recognition (CVPR Workshops 2014, pp. 806–813). Using the public OverFeat network trained for ILSVRC-2013 object classification, they extract a 4096-dimensional layer representation and apply a linear SVM (or L2 distance for retrieval) across object classification, scene recognition, fine-grained recognition, attribute detection, and image retrieval—tasks chosen to move progressively farther from the original training task and data. They report consistently strong results relative to highly tuned contemporaneous systems, with simple augmentation such as jittering. Together, DeCAF and the off-the-shelf CNN-features paper mark a community shift: ImageNet-trained ConvNets became default visual representations even when the downstream labels were scarce or differently distributed. Finetuning the whole stack later became common; it was not the only, or even the first, successful transfer pattern in this wave.

Distributional semantics at scale: word2vec

Language had a parallel representation economy. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean’s Efficient Estimation of Word Representations in Vector Space (ICLR 2013 workshop; arXiv:1301.3781) proposes two architectures—continuous bag-of-words (CBOW) and continuous skip-gram—for learning continuous word vectors from very large corpora at low computational cost. Quality is measured on word-similarity and syntactic/semantic analogy-style tests; they report high-quality vectors from a 1.6-billion-word dataset in less than a day of training. The follow-up Distributed Representations of Words and Phrases and their Compositionality (Mikolov, Sutskever, Chen, Corrado, and Dean; Advances in Neural Information Processing Systems 26, 2013, pp. 3111–3119; arXiv:1310.4546) develops the skip-gram model further—subsampling of frequent words, negative sampling, and phrase vectors—and popularizes the linear-regularity folklore (e.g., vector arithmetic approximating analogical relations). Neither paper is a deep bidirectional transformer. Both instantiate an older distributional idea—you shall know a word by the company it keeps—as scalable neural prediction: predict context from a word, or a word from context, on unlabeled text, and reuse the resulting vectors as features. That is pretraining of embeddings, not of a full task model, but it cemented unlabeled text as a representation resource for downstream supervised NLP.

BERT: masked LM, next-sentence, and the pretrain/finetune recipe

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova’s BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (NAACL-HLT 2019, pp. 4171–4186; arXiv:1810.04805, 2018) is the paper that crystallized pretrain then finetune as a default NLP recipe for many labs. BERT pretrains a deep bidirectional Transformer encoder on unlabeled text with two objectives: a masked language model (randomly mask tokens—classically 15%—and predict the originals from both left and right context) and next-sentence prediction (decide whether sentence B follows sentence A). At finetuning time, the same architecture is adapted with a small task-specific output layer for sentence- and token-level tasks; the paper reports strong gains on GLUE, SQuAD, and related benchmarks. The contribution, carefully stated, is not “inventing transfer learning.” BERT itself situates prior unsupervised feature-based and fine-tuning lines (including word embeddings, ELMo-style contextual features, and OpenAI GPT-style left-to-right LM pretraining). What it packaged for the field was a unified bidirectional pretraining objective plus a minimal-architecture finetuning path that worked across many tasks without heavy task-specific engineering.

The other half briefly: autoregressive generative pretraining

The same year BERT’s preprint circulated, the complementary half of the later BERT/GPT split was already public. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever’s Improving Language Understanding by Generative Pre-Training (OpenAI, 2018) trains a Transformer language model on unlabeled text with a standard left-to-right generative objective, then discriminatively finetunes on downstream tasks. The accompanying OpenAI note frames the result as unsupervised pretraining plus supervised adaptation on a suite of entailment, similarity, reading-comprehension, and related benchmarks. BERT’s masked, bidirectional objective and GPT’s autoregressive objective are different answers to the same scarcity problem—learn from unlabeled text first—rather than a single invention. Full scaling of transformers and language models belongs to the next chapter; the point here is only that pretrain/finetune in NLP was already a two-track recipe by 2018–2019.

Self-supervision in vision: one contrastive line

Self-supervision is not “unsupervised magic with no task.” It replaces human labels with a pretext task whose labels are free to construct from the data itself—masks, next tokens, or, in vision, agreement between differently augmented views. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton’s A Simple Framework for Contrastive Learning of Visual Representations (ICML 2020; PMLR 119, pp. 1597–1607; arXiv:2002.05709), known as SimCLR, is a clean, verifiable instance. The framework maximizes agreement between two augmented views of the same image under a contrastive loss, treating other in-batch views as negatives, without requiring a specialized architecture or a memory bank. They emphasize composition of data augmentations, a learnable nonlinear projection head between the representation and the contrastive loss, and the benefit of large batches and long training relative to supervised baselines. A linear classifier on SimCLR representations reaches 76.5% ImageNet top-1 in their reported setup, matching a supervised ResNet-50 under that protocol; with finetuning on only 1% of labels they report 85.8% top-5. The historical load-bearing claim is modest: by 2020, vision had a scalable self-supervised path to transferable features that did not depend on ImageNet class labels for the representation-learning stage—completing a circle from 2014’s supervised-ImageNet features toward unlabeled visual corpora.

What the modern stack actually is

Put as a single argument: labeled supervision for the target task remains scarce relative to raw data. Layerwise RBM pretraining (2006) answered a deep-net optimization problem. ImageNet transfer (2014) answered a feature reuse problem with supervised source labels. Word2vec answered a distributional embedding problem on unlabeled text. BERT and GPT-style models answered a contextual representation problem with language-model (masked or autoregressive) pretraining plus finetuning. Contrastive methods such as SimCLR answered a visual pretext problem that manufactures supervision from augmentations. The shared pattern is learn-once, adapt-many—not a single paper’s invention of “pretraining.”

Caveat

Three retellings especially distort this material.

First, “BERT invented pretraining / transfer learning.” Layerwise unsupervised pretraining, ImageNet feature transfer, and word-vector learning precede BERT by years; BERT’s own related-work section acknowledges prior unsupervised representation and fine-tuning lines. BERT popularized a particular bidirectional masked-LM + finetune package for NLP. It did not create the category.

Second, “self-supervision is unsupervised learning with no task.” Masked LM, next-sentence prediction, next-token prediction, and contrastive view agreement are tasks—automatically labeled ones. Calling them unsupervised is common shorthand; calling them taskless is false.

Third, “finetune is always better than frozen features.” The 2014 DeCAF and off-the-shelf CNN-features results often used fixed or lightly adapted deep activations with simple classifiers. Finetuning later became a powerful default, especially for large language models, but the transfer literature’s early wins were frequently frozen-feature wins. Treating finetune-everything as historically necessary erases that evidence.

Sources

Primary