Bitware News
History · Ch. 102026-09-22

The Deep Learning Turn

What moved in 2012 was not a sudden invention of deep nets: ImageNet-scale labeled data, GPU training, ReLU, and dropout made a large ConvNet win a public, comparable benchmark—AlexNet as the ImageNet moment, not the first deep network.

HistoryInfraEval

Primary source: Krizhevsky, Sutskever & Hinton, NIPS 2012 (AlexNet): https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf

Earlier chapters treated backpropagation as a community tool for multilayer supervised learning and traced how sequence models became trainable transduction problems before self-attention. Those accounts still left open a different historical question: why, around 2012, did deep convolutional networks suddenly become the default story of machine learning’s return to neural nets? The slogan that “AlexNet invented deep learning” is folklore. Deep and convolutional networks already had a paper trail; so did layerwise unsupervised pretraining. What actually moved was a conjunction of ingredients that made a large ConvNet win a public, comparable vision benchmark at ImageNet scale: a large labeled dataset and challenge harness, commodity GPU training of a wide network, rectified linear units that sped gradient descent relative to saturating nonlinearities, and dropout as a regularizer against co-adaptation. This chapter states what each ingredient did, situates the 2012 result as a turn rather than a birth, and leaves full pretrain/finetune stacks and later sequence architectures to subsequent chapters.

Prior deep and convolutional lines: a turn, not a birth

Convolutional nets for recognition are older than 2012. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner’s Gradient-Based Learning Applied to Document Recognition (Proceedings of the IEEE, vol. 86, no. 11, November 1998, pp. 2278–2324) is the canonical long-form statement of that line: gradient-based learning, convolutional architectures (including LeNet-5), and end-to-end systems for document and handwriting recognition, with the architectural inductive biases—local receptive fields, weight sharing, pooling—already explicit. Related earlier work from the same group on zip-code digits appeared in the late 1980s; the point for this chapter is not to re-litigate LeNet priority, but to record that deep-ish convolutional supervised learning was a working research program long before ImageNet.

A different “deep” strand in the mid-2000s emphasized layerwise unsupervised pretraining as a way to initialize deep nets that were otherwise hard to train purely supervised. Geoffrey E. Hinton and Ruslan R. Salakhutdinov’s Reducing the Dimensionality of Data with Neural Networks (Science, vol. 313, no. 5786, 28 July 2006, pp. 504–507) showed how a deep autoencoder could be pretrained as a stack of restricted Boltzmann machines and then fine-tuned, producing low-dimensional codes that preserved structure better than principal components analysis on several datasets. That paper is often cited as a landmark for deep belief nets and greedy layerwise training. It is not AlexNet: it is not a large ConvNet trained end-to-end on a million labeled natural images for a 1000-way classification challenge. The honest claim for 2012 is therefore narrower than a founding myth. Deep nets and ConvNets existed; what changed was the demonstration that a large, purely supervised ConvNet could dominate a community-scale visual recognition benchmark when data, compute, nonlinearities, and regularization lined up.

ImageNet and ILSVRC: scale plus a comparable eval harness

The dataset half of that conjunction is ImageNet. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei introduced it in ImageNet: A Large-Scale Hierarchical Image Database at CVPR 2009 (pp. 248–255). Built on the WordNet synset hierarchy and populated with web images cleaned via Amazon Mechanical Turk, ImageNet aimed at an ontology-scale resource—orders of magnitude beyond the tens-of-thousands image sets then common in object recognition. The CVPR paper reports an early snapshot with 12 subtrees, 5,247 synsets, and about 3.2 million images, and argues for scale, diversity, accuracy, and hierarchical structure as research infrastructure.

Scale alone does not make a scientific event. The evaluation harness that made results comparable across labs was the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), run annually from 2010. Olga Russakovsky, Jia Deng, Hao Su, and colleagues’ retrospective ImageNet Large Scale Visual Recognition Challenge (International Journal of Computer Vision, vol. 115, 2015, pp. 211–252) documents the dataset construction, the classification and detection tasks, and the community dynamics of the challenge. For classification, the familiar ILSVRC subset uses roughly 1.2 million training images across 1,000 categories, with top-1 and top-5 error as the headline metrics. That public, yearly comparison—not merely the existence of many labeled photos—is what turned a large ConvNet win into a field-wide signal.

ReLU: two papers, not one slogan

AlexNet’s nonlinearities were rectified linear units, (f(x) = \max(0, x)). The paper trail for ReLU in this period should not be collapsed into a single citation.

Vinod Nair and Geoffrey E. Hinton’s Rectified Linear Units Improve Restricted Boltzmann Machines (ICML 2010, pp. 807–814) introduces noisy rectified linear units as an efficient approximation to stepped sigmoid units in RBMs. Relative to binary hidden units, they report better features for object recognition on NORB and face verification on Labeled Faces in the Wild, and note that rectified units preserve relative intensity information across layers. The setting is generative / RBM, not a large supervised ConvNet on ImageNet; the nonlinearity that later became ubiquitous is already named and motivated there.

Xavier Glorot, Antoine Bordes, and Yoshua Bengio’s Deep Sparse Rectifier Neural Networks (AISTATS 2011; Proceedings of Machine Learning Research 15, pp. 315–323) studies rectifying activations in deep supervised multilayer networks. They argue that rectifiers yield sparse representations with true zeros, work despite the hard nonlinearity at zero, and—crucially for the later supervised story—can reach strong performance on large labeled datasets without unsupervised pretraining. That claim is a direct bridge to AlexNet’s “purely supervised” training narrative. AlexNet itself cites Nair and Hinton for the ReLU name and reports that deep ConvNets with ReLUs train several times faster than tanh equivalents on CIFAR-10—an optimization-speed claim about fitting large models, not a claim that ReLU was invented in 2012.

Dropout: paper trail before and beside the ImageNet win

Overfitting a 60-million-parameter net, even with 1.2 million labeled examples, was a first-order problem. The regularizer AlexNet highlights is dropout: randomly setting hidden-unit outputs to zero during training (typically with probability 0.5) so that units cannot rely on particular co-adapted partners, then using a weight-scaled full network at test time.

The contemporaneous technical statement is Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov’s Improving neural networks by preventing co-adaptation of feature detectors (arXiv:1207.0580, 3 July 2012). The expanded journal version is Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov’s Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Journal of Machine Learning Research 15, no. 56, 2014, pp. 1929–1958). AlexNet cites the 2012 arXiv preprint and applies dropout in its first two fully connected layers, noting that without it the network overfits substantially and that dropout roughly doubles the iterations to convergence. Treating dropout as a 2012-only miracle that appeared from nowhere inside the ImageNet paper erases that paper trail; treating the ImageNet win as “just dropout” erases data scale, GPUs, architecture, and ReLU.

AlexNet 2012: the ImageNet moment

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton’s ImageNet Classification with Deep Convolutional Neural Networks appeared in Advances in Neural Information Processing Systems 25 (NIPS 2012), pp. 1097–1105. The network—later nicknamed AlexNet—has five convolutional layers (with max-pooling after some) and three fully connected layers ending in a 1000-way softmax; about 60 million parameters and 650,000 neurons. On the ILSVRC-2010 test set they report top-1 / top-5 error of 37.5% / 17.0%, well below then-published sparse-coding and Fisher-vector baselines. In ILSVRC-2012, a multi-model variant achieved a winning top-5 test error of 15.3%, against 26.2% for the second-best entry.

What the paper actually says about GPUs matters, because folklore often invents FLOP legends. A single GTX 580 had 3 GB of memory, which limited network size; the authors spread the net across two NVIDIA GTX 580 3GB GPUs, with kernels partitioned across devices and communication only in certain layers. They report training for roughly ninety epochs over 1.2 million images in five to six days on that two-GPU setup, and they emphasize a highly optimized GPU implementation of 2D convolution (publicly released as cuda-convnet). GPUs are infrastructure that made training a network of this width and depth practical on the available hardware. They are not, by themselves, an explanation of the accuracy jump: without ImageNet-scale labels, ReLU-speeded optimization, data augmentation, dropout, and the convolutional inductive bias, faster matrix multiplies would not have produced the ILSVRC signal.

Other ingredients in the same paper are easy to under-count. Data augmentation—random crops and horizontal flips from 256×256 images down to 224×224 training patches, plus PCA-based RGB intensity perturbation—artificially enlarged the effective training distribution. Local response normalization and overlapping pooling are reported as incremental accuracy contributors. Depth is treated as load-bearing: removing any convolutional layer hurt performance. The learning recipe is ordinary SGD with momentum 0.9, weight decay 0.0005, batch size 128, and a manually dropped learning rate—infrastructure and recipe together, not a mysterious new optimizer.

What each ingredient actually did

Put briefly: ImageNet and ILSVRC supplied labeled scale and a shared scoreboard. GPUs and an efficient convolution implementation supplied wall-clock feasibility for a net too large for one 3 GB device. ReLU (via the Nair–Hinton and Glorot–Bordes–Bengio lines AlexNet builds on) supplied faster supervised fitting than saturating tanh/logistic units for deep nets. Dropout supplied regularization against co-adaptation in the huge fully connected layers. The convolutional architecture supplied inductive bias that made those parameters spendable on images. The 2012 event is the conjunction under a public benchmark—not the first deep network, and not a single magic trick.

A forward pointer only: later chapters take up how large supervised (and then pretrained) models became reusable features and how sequence architectures absorbed related scaling lessons. Scaling laws as such are out of scope here.

Caveat

Three retellings especially distort this material.

First, “AlexNet invented deep learning / was the first deep network.” LeCun-style ConvNets and Hinton–Salakhutdinov-style deep pretrained nets precede 2012. AlexNet is the ImageNet-scale supervised ConvNet moment that shifted community baselines, not the birth of depth.

Second, “GPUs alone explain 2012.” The AlexNet paper is explicit about two GTX 580s and multi-day training; that is real infrastructure history. Accuracy also required ImageNet/ILSVRC, ReLU, dropout, augmentation, and architecture. Faster GPUs without those pieces would not have written the same chapter.

Third, “dropout was a 2012-only miracle.” The method is documented in Hinton et al.’s July 2012 arXiv preprint and expanded in Srivastava et al. 2014 JMLR; AlexNet applies and cites it. The folklore that forgets the paper trail, and the folklore that reduces the whole turn to dropout, are both wrong.

Sources

Primary