Bitware News
History · Ch. 082026-09-20

How We Train: GD to Adam

Gradient descent, stochastic approximation, momentum, regularization, and Adam form a cumulative training-methods line—engineering and theory, not a folklore of sudden deep-learning inventions.

History

Primary source: Kingma & Ba, Adam (ICLR 2015): https://doi.org/10.48550/arXiv.1412.6980

Earlier chapters treated backpropagation as the method that made multilayer supervised learning a shared tool, and the 1990s statistical turn—VC theory, soft-margin SVMs, kernels—as a capacity-controlled alternative with clearer generalization rhetoric. Neither chapter finished the training-methods story. Once a network (or any differentiable model) yields a scalar loss and a gradient, one still has to choose how to walk the parameter space: batch or stochastic updates, whether to accumulate velocity, how to penalize complexity, and whether to adapt stepsize per coordinate. That optimizer stack is method, not folklore. Gradient descent, Robbins–Monro stochastic approximation, Polyak’s heavy ball and Nesterov’s accelerated gradient, Tikhonov-style and weight-decay regularization, and Kingma and Ba’s Adam are a cumulative engineering-and-theory line. This chapter is the training-methods spine promised in the series plan—not a complete convex-optimization textbook, and not the AlexNet or transformer stories reserved for later chapters.

Gradient descent: Cauchy 1847, carefully

There is no single verified “invention of gradient descent” date that one can pin to modern machine learning without anachronism. What can be cited cleanly is Augustin-Louis Cauchy’s short note Méthode générale pour la résolution des systèmes d’équations simultanées, presented to the Académie des Sciences and published in the Comptes rendus of 18 October 1847 (tome 25, pp. 536–538). Cauchy’s motivation is solving simultaneous equations—astronomical orbit elements among the intended applications—not training neural nets. The constructive idea is nonetheless recognizably steepest-descent style: given a nonnegative function (u = f(x,y,z,\ldots)), move the unknowns opposite the partial derivatives (the gradient), choosing a stepsize so that (u) decreases; systems (u=v=w=\cdots=0) are reduced by minimizing a sum of squares. English translations and historical commentary (for example the Waterloo classics reprint of the note) make that geometry explicit.

What this history does not claim: that Cauchy “invented SGD,” that 1847 is the birth year of deep learning, or that every later first-order method is a footnote to that page. Nineteenth-century analysis also contains other descent and variational ideas; attributing a single founding myth to one paper substitutes genealogy for method. For this series, Cauchy 1847 is a documented early statement of iterative minimization along the negative gradient. Modern batch gradient descent on an empirical risk is the same first-order idea specialized to a finite sum of losses.

Stochastic approximation: Robbins and Monro 1951

Batch gradient descent evaluates (or approximates) the full gradient of the training objective. Large, redundant datasets make that wasteful: successive halves of the data often give nearly the same direction. The mathematical ancestor of stochastic gradient methods is older than the 2010s deep-learning boom. In September 1951, Herbert Robbins and Sutton Monro published A Stochastic Approximation Method in The Annals of Mathematical Statistics (vol. 22, no. 3, pp. 400–407; DOI 10.1214/aoms/1177729586).

Their setting is not neural-net training. Let (M(x)) be the expected response at level (x) of an experiment; (M) is monotone but unknown, and one wants the root of (M(x)=\alpha). Robbins and Monro give a recursive scheme that chooses successive experimental levels (x_n) from noisy observations so that (x_n) converges in probability to the root under stated conditions. The procedure is stochastic approximation: update from noisy, sequentially observed feedback rather than from an exact gradient of a known objective.

That paper is SGD’s mathematical ancestor in the careful sense: later stochastic gradient descent on empirical risk—updating parameters from a minibatch or a single example—is a first-order optimization practice that lives in the same conceptual family as solving equations from noisy observations with decreasing steps. Treating “SGD” as a deep-learning invention of the 2010s is folklore. Minibatch SGD as a practical neural-net workhorse is much later engineering; the stochastic-approximation lineage is mid-twentieth-century statistics.

Momentum: heavy ball and Nesterov acceleration

Plain gradient steps struggle on elongated valleys: the gradient is large along tightly curved directions and small along the long, gentle axis, inviting oscillation or crawl. Momentum methods add inertia so that the update remembers previous steps.

In 1964, Boris T. Polyak published Some methods of speeding up the convergence of iteration methods (Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki 4, no. 5, pp. 791–803; English: U.S.S.R. Computational Mathematics and Mathematical Physics 4, no. 5, pp. 1–17; DOI 10.1016/0041-5553(64)90137-5). Among the multi-step schemes is what later literature calls the heavy-ball method: a discrete analogue of a heavy sphere with friction in a potential field. The update includes a term (\beta(x_n - x_{n-1})) that carries inertia, so motion need not align with the instantaneous antigradient. Polyak discusses parameter choice, possible non-monotonic progress, and empirical speedups relative to plain gradient iteration. This is classical optimization—functional equations and differentiable minimization in Banach/Hilbert settings—not a neural-net trick invented in the connectionist 1980s.

A distinct acceleration result appears in 1983. Yurii Nesterov’s A method of solving a convex programming problem with convergence rate (O(1/k^2)) (Doklady Akademii Nauk SSSR 269, no. 3, pp. 543–547; English: Soviet Mathematics Doklady 27, pp. 372–376) constructs a non-relaxational minimizing sequence with an unimprovable rate, for the class considered, of order (1/k^2) for smooth convex problems. The scheme that machine-learning practice later nicknames “Nesterov momentum” evaluates the gradient at a lookahead point (extrapolate first, then correct)—mechanically different from Polyak’s heavy ball, which accumulates velocity at the current point. Collapsing both into “momentum as a neural-net heuristic with no earlier optimization literature” is folklore. Rumelhart, Hinton, and Williams’s 1986 backpropagation presentations already mention a practical momentum term; the optimization literature’s heavy-ball and accelerated-gradient analyses are older and broader.

Regularization: Tikhonov, weight decay, and a later clarification

Optimization without capacity control overfits. The classical analytic statement of regularization for ill-posed problems is A. N. Tikhonov’s 1963 note On the solution of ill-posed problems and the method of regularization (Doklady Akademii Nauk SSSR 151, no. 3, pp. 501–504; English: Solution of incorrectly formulated problems and the regularization method, Soviet Mathematics Doklady 4, 1963, pp. 1035–1038). Tikhonov’s program stabilizes inversion by trading data fit against a penalty on the solution’s “complexity” (in the original setting, norms that encode smoothness or other prior structure). Machine-learning practice inherits the same trade-off language: minimize empirical loss plus a regularizer.

In the neural-net literature, a standard regularizer is weight decay—penalizing large weights so that unused directions shrink. Anders Krogh and John A. Hertz’s A Simple Weight Decay Can Improve Generalization (NIPS 1991; Advances in Neural Information Processing Systems 4, pp. 950–957) analyzes why. Adding (\frac{\lambda}{2}\sum_i w_i^2) to the cost yields an update that subtracts (\lambda w_i) alongside the usual error gradient. For linear networks they prove two effects: suppression of irrelevant weight components (preferring small solutions that still fit) and, with an appropriate (\lambda), suppression of some static target noise. They extend the discussion to hidden layers and nonlinear units and confirm the theory on NetTalk-style simulations. Calling weight decay a deep-learning invention is again folklore; it is 1990s (and earlier informal) neural-net practice continuous with Tikhonov-style penalization.

A brief later clarification matters for honesty about terminology. For ordinary SGD, L2 regularization (adding (\lambda|w|^2) to the loss) and multiplicative weight decay can be matched by rescaling (\lambda) with the learning rate. For adaptive methods such as Adam they are not equivalent: L2 couples the penalty into the adaptive gradient scaling, so parameters with large historical gradients are regularized differently than under true weight decay. Ilya Loshchilov and Frank Hutter’s Decoupled Weight Decay Regularization (arXiv:1711.05101, 2017; ICLR 2019) make that inequivalence explicit and propose AdamW—apply weight decay as a separate multiplicative shrink, decoupled from Adam’s moment-based step. This chapter records the clarification; it does not retell the full post-2017 optimizer zoo.

Adaptive methods and Adam

Coordinate-wise adaptation addresses a different pathology: gradient magnitudes that differ wildly across parameters (sparse features, depth, fan-in). John Duchi, Elad Hazan, and Yoram Singer’s Adaptive Subgradient Methods for Online Learning and Stochastic Optimization (Journal of Machine Learning Research 12, 2011, pp. 2121–2159)—AdaGrad—accumulates squared gradients and scales steps inversely with that history, giving rare but informative features larger effective rates. Geoffrey Hinton’s Coursera lectures Neural Networks for Machine Learning (lecture 6e, with slides circulated as Tieleman and Hinton’s RMSProp notes, 2012) introduce RMSProp: keep an exponentially decaying moving average of squared gradients and divide the update by its root mean square, a minibatch-stable relative of RPROP-style magnitude normalization. Both are immediate ancestors of the method that became the default adaptive baseline.

In December 2014, Diederik P. Kingma and Jimmy Lei Ba posted Adam: A Method for Stochastic Optimization (arXiv:1412.6980); the work appeared as a conference paper at ICLR 2015. Adam maintains exponential moving averages of the gradient (m_t) (first moment) and of the squared gradient (v_t) (second raw moment), with decay hyperparameters (\beta_1) and (\beta_2). Because those averages are initialized at zero, early estimates are biased toward zero; Adam applies bias correction, [ \hat{m}_t = \frac{m_t}{1-\beta_1^t},\qquad \hat{v}t = \frac{v_t}{1-\beta_2^t}, ] and steps [ \theta_t \leftarrow \theta{t-1} - \alpha,\hat{m}_t\big/\big(\sqrt{\hat{v}_t}+\epsilon\big). ] The authors explicitly combine AdaGrad’s strength on sparse gradients with RMSProp’s suitability for non-stationary objectives, argue invariance to diagonal gradient rescaling, and report favorable comparisons on large stochastic problems. The name is “adaptive moment estimation,” not a claim that no other optimizer works.

Adam’s popularity does not erase SGD with momentum, carefully tuned schedules, or problem-dependent choice. The folklore that “Adam is the only optimizer that works” confuses a versatile default with a theorem. The verified claim is narrower: Adam is a clearly specified first-order method with moment estimates and bias correction, sitting at the end of a documented line from Cauchy’s descent geometry through stochastic approximation, momentum, regularization, and adaptive scaling.

Caveat

Three retellings especially distort this material.

First, “SGD was invented in the 2010s for deep learning.” Minibatch stochastic gradient became a practical deep-learning workhorse then; Robbins and Monro’s 1951 stochastic approximation already formalized recursive updates from noisy observations aimed at a root or optimum. The engineering practice and the mathematical ancestor are different layers of the same stack.

Second, “momentum is a neural-net trick with no earlier optimization literature.” Polyak’s 1964 heavy-ball method and Nesterov’s 1983 (O(1/k^2)) acceleration are optimization results. Connectionist papers used momentum as a practical accelerator; they did not originate the idea of inertial or accelerated first-order steps.

Third, “Adam is the only optimizer that works.” Adam is a strong, widely used adaptive baseline with an explicit moment-and-bias-correction design. SGD with momentum, schedule design, and (after 2017–2019) decoupled weight decay remain competitive on many tasks. Treating Adam as metaphysics rather than method is folklore.

Sources

Primary