Optimization: Adam
Claude prompt
This page was generated by Claude Code from the following prompt (with additional prompting):
Call the next lecture adam. My plan is to introduce momentum, RMSprop and Adam since they are important for machine learning. Can we produce somethin close to figure 4.58 https://mdobook.github.io/html/unconstrained/#sec-lbfgs, but with adam added as a demo
make the demo much more concise: drop rmsprop and momentum and reduce the text and number of sliders … for adam notation, use the notation in this book: https://algorithmsbook.com/optimization/
This reproduces Figures 4.57 and 4.58 from Example 4.18 of Engineering Design Optimization by Martins and Ning (CC BY-NC-SA), with Adam added. Rosenbrock's function \(f = (1 - x)^2 + 100\,(y - x^2)^2\) minimized from \(\vec{x}_0 = (-1.2,\, 1.0)\) until \(\|\nabla f\| \le 10^{-6}\), by four methods that share the same Armijo backtracking line search (\(\mu_1 = 10^{-4}\), \(\rho = 0.5\)), plus Adam, which takes one gradient per iteration and no line search. In the notation of Algorithms for Optimization (Eqs. 5.30–5.34, with subscripts for iterates), with \(\vec{g}_k = \nabla f(\vec{x}_k)\), \(\gamma_v = 0.9\), \(\gamma_s = 0.999\), \(\epsilon = 10^{-8}\), and elementwise vector operations:
\[ \begin{aligned} \text{biased decaying momentum:} \quad & \vec{v}_{k+1} = \gamma_v \vec{v}_k + (1 - \gamma_v)\, \vec{g}_k \\ \text{biased decaying sq. gradient:} \quad & \vec{s}_{k+1} = \gamma_s \vec{s}_k + (1 - \gamma_s)\, \big(\vec{g}_k \odot \vec{g}_k\big) \\ \text{corrected decaying momentum:} \quad & \hat{\vec{v}}_{k+1} = \vec{v}_{k+1} / (1 - \gamma_v^{\,k+1}) \\ \text{corrected decaying sq. gradient:} \quad & \hat{\vec{s}}_{k+1} = \vec{s}_{k+1} / (1 - \gamma_s^{\,k+1}) \\ \text{next iterate:} \quad & \vec{x}_{k+1} = \vec{x}_k - \frac{\alpha}{\epsilon + \sqrt{\hat{\vec{s}}_{k+1}}} \odot \hat{\vec{v}}_{k+1} \end{aligned} \]Click on the contour plot to move the starting point. On the quadratic, Newton needs one step and BFGS a handful, while Adam still takes hundreds, since its step is about \(\alpha\) per coordinate no matter how simple the function. Each line-search iteration hides several function and gradient evaluations, so the plot against evaluations is the fairer comparison; Adam's step in each coordinate is about \(\alpha\) whatever the gradient's size, which is what makes it robust to the scaling and noise of neural network training, but its convergence is at best linear.