Optimization: Stationary Points and the Hessian
Claude prompt
This page was generated by Claude Code from the following prompts (with additional prompting):
Ok, let's make a demo. [...] Second will be visualizing 2d functions with a contour and surface plot. Give me a slider (or really 2 I guess) to control whether the Hessian is pos def, neg def, or makes a saddle point.
For lecture 130, I only got through talking about Hessians. I didn't talk about any solution techniques. So let's make that one just the demo content about the Hessian. Also give a 3D surface plot of the function in the handout from that lecture.
1. What the Hessian looks like near a stationary point
Near a point where the gradient is zero, every smooth function looks like a quadratic built from its Hessian \(H\), the matrix of second derivatives: \[ f(x, y) \approx \tfrac{1}{2} \begin{bmatrix} x & y \end{bmatrix} H \begin{bmatrix} x \\ y \end{bmatrix}, \qquad H = \begin{bmatrix} \dfrac{\partial^2 f}{\partial x^2} & \dfrac{\partial^2 f}{\partial x\,\partial y} \\[1.1em] \dfrac{\partial^2 f}{\partial x\,\partial y} & \dfrac{\partial^2 f}{\partial y^2} \end{bmatrix}. \] The quadratic below is set up through the eigenvalues of \(H\), which are the curvatures along its two principal directions, and the angle of the first principal direction. The signs of the eigenvalues decide what kind of point the origin is.
The dashed lines on the contour plot are the eigenvectors of \(H\). Along each one the function is a parabola with curvature equal to that eigenvalue. Both positive: a bowl, the origin is a minimum, \(H\) is positive definite. Both negative: a dome, a maximum, negative definite. One of each: a saddle, which is a maximum along one line and a minimum along the other, and \(H\) is indefinite. Without computing eigenvalues, the same test is \[ \det H = \frac{\partial^2 f}{\partial x^2}\,\frac{\partial^2 f}{\partial y^2} - \left(\frac{\partial^2 f}{\partial x\,\partial y}\right)^2 : \] a saddle has \(\det H < 0\), and when \(\det H > 0\) the sign of \(\partial^2 f / \partial x^2\) says whether it is a minimum or a maximum. The surface can be dragged to rotate it.
2. The function from the in-class assignment
The second exercise on the handout asked you to classify the stationary points of \[ f(x, y) = \frac{x^3}{3} - x + \frac{y^3}{3} - y \] from its contour plot alone. Here is the same function as a surface (drag to rotate it), with the four kinds of point from part 1 in one place. Its gradient and Hessian are \[ \nabla f = \begin{bmatrix} x^2 - 1 \\ y^2 - 1 \end{bmatrix}, \qquad H = \begin{bmatrix} 2x & 0 \\ 0 & 2y \end{bmatrix}, \] so the first-order necessary condition \(\nabla f = 0\) holds at exactly four points, \((\pm 1, \pm 1)\). Pick one to see its Hessian.
Because the Hessian here happens to be diagonal, its eigenvalues are just the diagonal entries \(2x\) and \(2y\). At \((1,1)\) both are positive, so \(H\) is positive definite and the point is a local minimum (a bowl). At \((-1,-1)\) both are negative: a local maximum (a dome). At the other two the signs differ, so \(H\) is indefinite and the surface is a saddle, curving up along one axis and down along the other. Notice that \((1,1)\) is only a local minimum: \(f\) keeps decreasing toward the corner of the plotted region, and it is unbounded below.