Part I · 第 3 章多元与向量微积分 · 3.5Part I · Ch. 3 Multivariable & Vector Calculus · 3.5

Hessian 与曲率Hessian & Curvature

梯度告诉你脚下的坡度，却不告诉你坡怎样变化：前面是碗底，是山顶，还是一匹马鞍？这需要二阶导数。把所有二阶偏导排成一个对称矩阵，就是 Hessian。它的特征值是各个方向上的曲率，特征向量是"最弯"和"最直"的方向。判断极小、极大与鞍点，设计牛顿法，估计状态估计里的不确定性椭圆，用的都是它。 The gradient tells you the slope underfoot, not how the slope changes: is a bowl floor ahead, a summit, or a saddle? That takes second derivatives. Arrange all second partials in a symmetric matrix and you get the Hessian. Its eigenvalues are the curvatures along the principal directions, its eigenvectors the "most curved" and "straightest" directions. It tells minima, maxima and saddles apart, drives Newton's method, and gives the uncertainty ellipse in state estimation.

约 25 分钟~25 min 前置：1.11 特征值 · 2.4 泰勒展开 · 3.2 梯度Prereqs: 1.11 Eigenvalues · 2.4 Taylor Expansion · 3.2 Gradient 5 个交互5 interactives

1为什么机器人需要它Why robots need this

只用梯度的优化器像蒙着眼下山：它知道脚下哪边是下坡，却不知道前面的地形。很多时候，这还不够：

步长怎么选？在陡峭的方向要迈小步，平缓的方向要迈大步。"陡"和"缓"指的是曲率，要看 Hessian 的特征值。条件数 $\kappa=\lambda_{\max}/\lambda_{\min}$ 大的问题，梯度下降会慢得让人崩溃（6.3 节）。
牛顿法与高斯–牛顿：利用曲率，一步跳到局部二次模型的最低点。SLAM 的后端、光束法平差、轨迹优化的求解器本质上都在这么做（6.6、6.9 节）。
驻点是什么点：$\nabla f=\mathbf 0$ 的地方是极小、极大还是鞍点？训练神经网络和做轨迹优化时，鞍点远比局部极小常见。
不确定性：最大似然估计在最优点附近，负对数似然的 Hessian 就是信息矩阵，它的逆就是估计误差的协方差（4.4、4.8 节）。哪个方向定位得准、哪个方向漂，都写在 Hessian 里。

An optimizer that uses only the gradient is like walking downhill blindfolded: it knows which way is down underfoot but nothing about the terrain ahead. Often that is not enough:

How big a step? Steep directions need small steps and gentle ones large steps. "Steep" and "gentle" here mean curvature, i.e. the Hessian's eigenvalues. A problem with a large condition number $\kappa=\lambda_{\max}/\lambda_{\min}$ makes gradient descent painfully slow (Lesson 6.3).
Newton and Gauss–Newton: use the curvature to jump straight to the bottom of the local quadratic model. SLAM back-ends, bundle adjustment and trajectory-optimization solvers essentially do this (Lessons 6.6, 6.9).
What kind of stationary point? Is a point with $\nabla f=\mathbf 0$ a minimum, maximum or saddle? In neural-network training and trajectory optimization, saddles are far more common than local minima.
Uncertainty: near the optimum of a maximum-likelihood estimate, the Hessian of the negative log-likelihood is the information matrix, and its inverse is the covariance of the estimate error (Lessons 4.4, 4.8). Which direction is localized sharply and which drifts is written in the Hessian.

2直觉：二阶导数是"斜率的变化率"Intuition: the second derivative is the rate of change of the slope

先看一维。一阶导数 $f'$ 是切线的斜率，二阶导数 $f''$ 回答：沿着曲线走，切线本身转得多快。$f''>0$：斜率在增加，曲线向上弯，像碗；$f''<0$：斜率在减小，向下弯，像穹顶；$f''=0$：暂时不弯，比如拐点。

在点 $a$ 处，与曲线贴得最紧的抛物线是 $f(a)+f'(a)\delta+\tfrac12f''(a)\delta^2$（2.4 节的二阶泰勒展开）。它比切线多出的 $\tfrac12f''\delta^2$ 项，就是曲线弯离切线的部分。拖动圆点，换一个位置，再换一个窗口大小，比较切线（一阶）和抛物线（二阶）谁更贴近曲线。

Start in one dimension. The first derivative $f'$ is the slope of the tangent line; the second derivative $f''$ answers how fast the tangent itself turns as you walk along the curve. $f''>0$: the slope increases and the curve bends upward like a bowl; $f''<0$: the slope decreases and it bends downward like a dome; $f''=0$: momentarily straight, as at an inflection point.

At a point $a$, the parabola hugging the curve most tightly is $f(a)+f'(a)\delta+\tfrac12f''(a)\delta^2$ (the second-order Taylor expansion of Lesson 2.4). The extra $\tfrac12f''\delta^2$ beyond the tangent line is the part by which the curve bends away from it. Drag the dot, change the window size, and compare how well the tangent line (first order) and the parabola (second order) follow the curve.

FIG 1切线与密切抛物线Tangent line and osculating parabola

青色是曲线 $f$，琥珀色虚线是切线，紫色是二阶抛物线。灰色竖带是窗口 $\pm h$，读数是窗口边缘处两种近似的误差：把 $h$ 减半，切线误差约变为 $1/4$，抛物线误差约变为 $1/8$。 Cyan is the curve $f$, the dashed amber line is the tangent, and purple is the second-order parabola. The grey band is the window $\pm h$; the readout gives each approximation's error at the window edges: halve $h$ and the tangent's error falls to about $1/4$, the parabola's to about $1/8$.

找一个 $f''<-0.5$ 的位置（向下弯）。再找一个 $f''\approx0$ 的位置（拐点）：切线与抛物线此时有什么关系？Find a position with $f''<-0.5$ (bending down). Then find one with $f''\approx0$ (an inflection point): how do the tangent line and the parabola relate there?

3动手玩：Hessian 的二次型Play: the quadratic form of the Hessian

二维里，曲线的弯曲方向有很多个，所以需要一个矩阵。把所有二阶偏导排成表：

In two dimensions a surface can bend in many directions, hence the need for a matrix. Arrange all the second partials in a table:

$$ H = \begin{bmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy}\end{bmatrix} = \begin{bmatrix} a & b \\ b & c\end{bmatrix} \quad(\text{symmetric: } f_{xy}=f_{yx}). $$

最简单的曲面是纯二次函数 $q(\mathbf x)=\tfrac12\mathbf x^\top H\mathbf x$，它在原点处的 Hessian 就是 $H$。下图的三个滑块直接设定 $a,b,c$，左边是 $q$ 的等高线（原点处梯度为 0）。紫色手柄设定一个方向 $\mathbf u$；右上图把 $\mathbf u$ 方向的曲率 $\kappa(\mathbf u)=\mathbf u^\top H\mathbf u$ 画成朝向角 $\varphi$ 的函数，右下图是沿 $\mathbf u$ 切出的一维曲线 $q(t\mathbf u)=\tfrac12\kappa\,t^2$。红、绿两条直线是 $H$ 的两个主方向（特征向量）。

The simplest surface is a pure quadratic, $q(\mathbf x)=\tfrac12\mathbf x^\top H\mathbf x$, whose Hessian at the origin is exactly $H$. The three sliders in the figure set $a,b,c$ directly, and the left panel shows the contours of $q$ (the gradient vanishes at the origin). The purple handle sets a direction $\mathbf u$; the upper right plots the curvature along it, $\kappa(\mathbf u)=\mathbf u^\top H\mathbf u$, against the heading angle $\varphi$, and the lower right shows the one-dimensional cut $q(t\mathbf u)=\tfrac12\kappa\,t^2$. The red and green lines are the two principal directions (eigenvectors) of $H$.

FIG 2二次型实验室Quadratic-form lab

q(x) 的等高线CONTOURS OF q(x)

等高线颜色：青色偏低、紫色偏高。红色虚线（只在鞍面时出现）是 $q=0$ 的两条直线，也是曲率为 0 的方向：它们把"向上弯"的方向和"向下弯"的方向分开。红、绿实线是 $H$ 的特征向量方向，线旁的数字是对应的特征值（主曲率）。 Contour shading: cyan is low, purple is high. The red dashed lines (appearing only for saddles) are the two lines $q=0$, the directions of zero curvature, which separate directions that bend up from directions that bend down. The red and green solid lines are the eigenvector directions of $H$, and the numbers beside them are the eigenvalues (principal curvatures).

试一试Try this

造一个碗：两个特征值都为正。曲率随方向变化的曲线 $\kappa(\varphi)$ 此时全在 0 线上方。Make a bowl: both eigenvalues positive. The curve $\kappa(\varphi)$ then lies entirely above zero.
造一个鞍面：一正一负。曲线穿过 0 线，穿过的位置就是红色虚线的方向。Make a saddle: one positive, one negative. The curve crosses zero, at the directions of the red dashed lines.
造一个平底山谷：一个特征值大于 1，另一个几乎为 0（$|\lambda|<0.05$）。沿谷底方向走，$q$ 几乎不变。Make a flat-bottomed valley: one eigenvalue above 1, the other almost 0 ($|\lambda|<0.05$). Walking along the valley floor, $q$ hardly changes.
让两个主方向斜成 45°：$a\approx c$，$b\neq0$。轴向的偏导数看不出这条斜谷，Hessian 的非对角元素 $b$ 却记录了它。Make the principal directions tilt by 45°: $a\approx c$ and $b\neq0$. The axis-aligned partials do not reveal such a diagonal valley, but the off-diagonal entry $b$ records it.

核心洞见Key insight

沿单位方向 $\mathbf u$ 的曲率是二次型 $\kappa(\mathbf u)=\mathbf u^\top H\mathbf u$，最大值是最大特征值 $\lambda_{\max}$，最小值是 $\lambda_{\min}$，分别出现在对应的特征向量方向（1.11 节）。曲率的符号决定"碗"还是"穹顶"：特征值全正是碗（极小），全负是穹顶（极大），有正有负是鞍面。

The curvature along a unit direction $\mathbf u$ is the quadratic form $\kappa(\mathbf u)=\mathbf u^\top H\mathbf u$; its maximum is the largest eigenvalue $\lambda_{\max}$, its minimum $\lambda_{\min}$, attained along the corresponding eigenvectors (Lesson 1.11). The signs decide "bowl" or "dome": all positive eigenvalues make a bowl (minimum), all negative a dome (maximum), mixed signs a saddle.

4第二视角：任意函数在一点的二次模型，以及牛顿法Second views: the local quadratic model of any function, and Newton's method

任何光滑函数都"局部像二次函数"Every smooth function looks quadratic locally

真实的函数不是纯二次的，但在每个点 $\mathbf p$ 附近，它被二阶泰勒展开 $m(\boldsymbol\delta)=f+\nabla f^\top\boldsymbol\delta+\tfrac12\boldsymbol\delta^\top H\boldsymbol\delta$ 很好地近似。下图用三个真实函数演示：实线是 $f$ 的等高线，紫色虚线是二次模型的等高线（只在圆盘内绘制，半径可调）。选"一阶"，模型的等高线退化成互相平行的直线，那就是 3.1 节的线性近似；选"二阶"，它们变成椭圆或双曲线，和实线贴得更紧。

拖动黄点，右侧会报告 $H$ 的特征值与点的类型。Himmelblau 函数有 4 个极小、1 个极大、4 个鞍点，试着把它们都找出来。

Real functions are not pure quadratics, but near any point $\mathbf p$ they are well approximated by the second-order Taylor expansion $m(\boldsymbol\delta)=f+\nabla f^\top\boldsymbol\delta+\tfrac12\boldsymbol\delta^\top H\boldsymbol\delta$. The figure shows three real functions: solid lines are the contours of $f$, purple dashed lines the contours of the quadratic model (drawn only inside a disc of adjustable radius). Choose "1st order" and the model's contours degenerate into parallel straight lines, the linear approximation of Lesson 3.1; choose "2nd order" and they become ellipses or hyperbolas, hugging the solid ones much more closely.

Drag the amber point and the side panel reports the eigenvalues of $H$ and the type of point. Himmelblau's function has 4 minima, 1 maximum and 4 saddles: try to find them all.

FIG 3二阶泰勒模型Second-order Taylor model

实线：$f$ 的等高线；虚线：模型的等高线；淡色圆盘是模型的有效范围。黄点旁的红、绿线段是 $H$ 的两个主方向，长度与 $1/\sqrt{|\lambda|}$ 成正比（曲率越大，越短）。 Solid: contours of $f$; dashed: contours of the model; the pale disc is the model's range of validity. The red and green segments beside the amber point are the principal directions of $H$, with length proportional to $1/\sqrt{|\lambda|}$ (larger curvature, shorter segment).

在 Himmelblau 上找到一个极小点：梯度很小，Hessian 两个特征值都为正。On Himmelblau, find a minimum: small gradient and a Hessian with two positive eigenvalues.
再找一个鞍点：梯度很小，两个特征值一正一负。Then find a saddle: small gradient, one positive and one negative eigenvalue.
再找到那个极大点（两个特征值都为负）。And find the maximum (both eigenvalues negative).

牛顿法：一步跳到二次模型的谷底Newton's method: jump to the bottom of the quadratic model

既然 $f$ 局部像二次函数 $m$，不如直接求 $m$ 的最低点：令 $\nabla m=\nabla f+H\boldsymbol\delta=\mathbf 0$，得 $\boldsymbol\delta=-H^{-1}\nabla f$，这就是牛顿步。如果 $f$ 本身就是二次函数，一步就到最低点。相比之下，梯度下降（固定步长 $\eta$）每一步只用到一阶信息。当 $H$ 的特征值相差很大（条件数 $\kappa$ 大）时，最优固定步长下的梯度下降每步只能把误差压缩为原来的 $(\kappa-1)/(\kappa+1)$，$\kappa=30$ 时只有 0.935。

下图比较两者。在"二次碗"上调 $\kappa$；在"香蕉函数"（非二次）上，牛顿法也会遇到麻烦：若 $H$ 不是正定，牛顿步会指向鞍点或山顶，需要加阻尼 $H+\mu I$。

Since $f$ locally looks like a quadratic $m$, why not jump straight to the lowest point of $m$? Setting $\nabla m=\nabla f+H\boldsymbol\delta=\mathbf 0$ gives $\boldsymbol\delta=-H^{-1}\nabla f$, the Newton step. If $f$ is itself quadratic, one step reaches the minimum. Gradient descent with a fixed step $\eta$, by contrast, uses only first-order information. When the eigenvalues of $H$ differ greatly (a large condition number $\kappa$), even with the best fixed step each gradient-descent step only shrinks the error by a factor $(\kappa-1)/(\kappa+1)$: with $\kappa=30$ that is merely 0.935.

The figure compares the two. On the "Quadratic bowl" vary $\kappa$; on the "Banana" function (not quadratic), Newton's method has trouble too: where $H$ is not positive definite the Newton step heads for a saddle or a peak, and needs damping, $H+\mu I$.

FIG 4梯度下降 vs 牛顿法Gradient descent vs. Newton

黄点是起点（可拖动）。青色折线是梯度下降的前 $N$ 步，紫色折线是牛顿法的前 $N$ 步，红色菱形是真正的极小点。若 $H$ 在某步不是正定，该步的牛顿步用红色标出：它不保证是下降方向。为了留在画面内，单个牛顿步的长度被限制在 5 以内（二次碗上的起点不会触发这个限制）。 The amber dot is the start (draggable). The cyan polyline is the first $N$ gradient-descent steps, the purple one the first $N$ Newton steps, and the red diamond the true minimizer. If $H$ is not positive definite at some step, that Newton step is drawn in red: it is no longer guaranteed to be a descent direction. To stay on screen, a single Newton step is capped at length 5 (never triggered from any start on the quadratic bowl).

在"二次碗"上，把 $\kappa$ 拉到 20 以上、$N=1$：牛顿法一步到位，梯度下降还差多远？On the "Quadratic bowl", pull $\kappa$ above 20 with $N=1$: Newton lands exactly; how far is gradient descent?
切到"香蕉"，从 $(0,0)$ 一带出发，看牛顿法的第一步；再换一个让 $H$ 非正定的起点，用阻尼 $\mu$ 修好它。Switch to "Banana" and start around $(0,0)$ to see Newton's first step; then pick a start where $H$ is not positive definite and repair it with the damping $\mu$.

5再看数学The math

提示：把鼠标悬停在公式中带底色的符号上，FIG 2 中对应的元素会被高亮。Tip: hover the highlighted symbols in the formulas; the matching element in FIG 2 lights up.

定义与对称性Definition and symmetry

设 $f:\R^n\to\R$ 二阶连续可微。Hessian 是二阶偏导数的 $n\times n$ 矩阵，也是梯度的雅可比（3.3 节）：$H=J_{\nabla f}$。

Let $f:\R^n\to\R$ be twice continuously differentiable. The Hessian is the $n\times n$ matrix of second partials; it is also the Jacobian of the gradient (Lesson 3.3): $H=J_{\nabla f}$.

$$ H(\mathbf p) = \nabla^2 f(\mathbf p),\qquad H_{ij} = \frac{\partial^2 f}{\partial x_i\,\partial x_j}(\mathbf p) = H_{ji}. $$

对称性来自 3.1 节的 Schwarz 定理（二阶偏导连续时，求导次序可交换）。对称矩阵有实特征值和正交的特征向量（1.11 节的谱定理），这正是"主曲率"和"主方向"有意义的原因。

Symmetry is Schwarz's theorem from Lesson 3.1 (with continuous second partials the order of differentiation can be swapped). A symmetric matrix has real eigenvalues and orthogonal eigenvectors (the spectral theorem of Lesson 1.11), which is why "principal curvatures" and "principal directions" make sense.

二阶泰勒展开与曲率Second-order Taylor expansion and curvature

$$ f(\mathbf p+\boldsymbol\delta) \;=\; f(\mathbf p) \;+\; \nabla f(\mathbf p)^\top\boldsymbol\delta \;+\; \tfrac12\,\hl{quad}{\boldsymbol\delta^\top\,\hl{H}{\cv{H}}\,\boldsymbol\delta} \;+\; O(\|\boldsymbol\delta\|^3). $$

令 $\boldsymbol\delta=t\mathbf u$（$\|\mathbf u\|=1$），第三项变成 $\tfrac12t^2\,\mathbf u^\top H\mathbf u$，所以 $\mathbf u$ 方向上一元函数的二阶导数是

With $\boldsymbol\delta=t\mathbf u$ ($\|\mathbf u\|=1$) the third term becomes $\tfrac12t^2\,\mathbf u^\top H\mathbf u$, so the second derivative of the one-variable cut along $\mathbf u$ is

$$ \frac{d^2}{dt^2}f(\mathbf p+t\mathbf u)\Big|_{t=0} = \hl{kappa}{\cw{\mathbf u}^\top H\,\cw{\mathbf u}} \in [\hl{lam}{\lambda_{\min}},\,\lambda_{\max}]. $$

取值范围是 Rayleigh 商的性质：设 $H=Q\Lambda Q^\top$，把 $\mathbf u$ 展开到特征基下 $\mathbf u=\sum_ic_i\mathbf q_i$（$\sum c_i^2=1$），则 $\mathbf u^\top H\mathbf u=\sum_i\lambda_ic_i^2$，是特征值的加权平均。二维时，$\kappa(\varphi)=\tfrac{\lambda_1+\lambda_2}{2}+\tfrac{\lambda_1-\lambda_2}{2}\cos2(\varphi-\varphi_1)$，所以 FIG 2 的曲线是一条周期为 $\pi$ 的余弦曲线。

The range is a property of the Rayleigh quotient: write $H=Q\Lambda Q^\top$ and expand $\mathbf u=\sum_ic_i\mathbf q_i$ in the eigenbasis ($\sum c_i^2=1$); then $\mathbf u^\top H\mathbf u=\sum_i\lambda_ic_i^2$, a weighted average of the eigenvalues. In 2D, $\kappa(\varphi)=\tfrac{\lambda_1+\lambda_2}{2}+\tfrac{\lambda_1-\lambda_2}{2}\cos2(\varphi-\varphi_1)$, so the curve in FIG 2 is a cosine with period $\pi$.

判别驻点：二阶条件Classifying stationary points: second-order conditions

在 $\nabla f(\mathbf p)=\mathbf 0$ 的点，一阶项消失，函数的局部形状由 $\tfrac12\boldsymbol\delta^\top H\boldsymbol\delta$ 决定：

At a point where $\nabla f(\mathbf p)=\mathbf 0$ the first-order term vanishes and the local shape is set by $\tfrac12\boldsymbol\delta^\top H\boldsymbol\delta$:

$H$ 的特征值Eigenvalues of $H$	名称Name	驻点类型Stationary point
全为正all positive	正定 $H\succ0$positive definite $H\succ0$	严格局部极小strict local minimum
全为负all negative	负定 $H\prec0$negative definite $H\prec0$	严格局部极大strict local maximum
有正有负mixed signs	不定indefinite	鞍点saddle point
有 0（其余同号）some zero (rest same sign)	半定、奇异semi-definite, singular	不能下结论，需看更高阶inconclusive; need higher order

二维时无需算特征值：$\det H=ac-b^2=\lambda_1\lambda_2$，$\operatorname{tr}H=a+c=\lambda_1+\lambda_2$。$\det H<0$ 一定是鞍点；$\det H>0$ 时由 $a$ 的符号区分极小（$a>0$）与极大（$a<0$）。

In 2D you need not compute eigenvalues: $\det H=ac-b^2=\lambda_1\lambda_2$ and $\operatorname{tr}H=a+c=\lambda_1+\lambda_2$. $\det H<0$ is always a saddle; for $\det H>0$ the sign of $a$ separates a minimum ($a>0$) from a maximum ($a<0$).

牛顿法与梯度下降的收敛Newton versus gradient-descent convergence

$$ \underbrace{\boldsymbol\delta_{\text{N}}=-\big(H+\mu I\big)^{-1}\nabla f}_{\text{(damped) Newton step}},\qquad \underbrace{\boldsymbol\delta_{\text{GD}}=-\eta\,\nabla f}_{\text{gradient step}}. $$

对二次函数 $f=\tfrac12\mathbf x^\top H\mathbf x$（$H\succ0$），梯度下降的更新 $\mathbf x\leftarrow(I-\eta H)\mathbf x$ 在特征基下按每个分量独立缩放 $1-\eta\lambda_i$。要全部收敛，需要 $|1-\eta\lambda_i|<1$，即 $0<\eta<2/\lambda_{\max}$：曲率越大，能用的步长越小（6.3 节）。最优步长 $\eta=2/(\lambda_{\max}+\lambda_{\min})$ 给出收缩因子 $\rho=(\kappa-1)/(\kappa+1)$。牛顿法的 $H^{-1}$ 恰好抵消各方向的缩放，对二次函数一步到位；对一般函数，在极小点附近是二次收敛（误差平方级下降）。

For a quadratic $f=\tfrac12\mathbf x^\top H\mathbf x$ ($H\succ0$), the gradient-descent update $\mathbf x\leftarrow(I-\eta H)\mathbf x$ scales each eigencomponent independently by $1-\eta\lambda_i$. Convergence in all components needs $|1-\eta\lambda_i|<1$, i.e. $0<\eta<2/\lambda_{\max}$: the larger the curvature, the smaller the step you can afford (Lesson 6.3). The best step $\eta=2/(\lambda_{\max}+\lambda_{\min})$ gives the contraction factor $\rho=(\kappa-1)/(\kappa+1)$. Newton's $H^{-1}$ cancels the per-direction scaling exactly, reaching the minimum of a quadratic in one step; for general functions it converges quadratically (the error is squared at each step) near a minimum.

非正定时怎么办？$-H^{-1}\nabla f$ 在 $H$ 有负特征值时可能不是下降方向。常见补救：给 $H$ 加阻尼 $\mu I$（Levenberg–Marquardt，6.6 节），或使用信赖域（6.5 节）。

What if $H$ is not positive definite? When $H$ has a negative eigenvalue, $-H^{-1}\nabla f$ may not be a descent direction. Usual remedies: add damping $\mu I$ (Levenberg–Marquardt, Lesson 6.6) or use a trust region (Lesson 6.5).

与最小二乘、雅可比的关系Relation to least squares and the Jacobian

机器人里最常见的目标是 $f(\mathbf x)=\tfrac12\|\mathbf r(\mathbf x)\|^2$（残差平方和，如重投影误差、IK 误差）。设 $J$ 是残差 $\mathbf r$ 的雅可比，则

The commonest robotics objective is $f(\mathbf x)=\tfrac12\|\mathbf r(\mathbf x)\|^2$ (a sum of squared residuals such as reprojection or IK error). With $J$ the Jacobian of the residual $\mathbf r$,

$$ \nabla f = J^\top\mathbf r,\qquad \nabla^2 f \;=\; \underbrace{J^\top J}_{\text{Gauss–Newton}} \;+\; \sum_i r_i\,\nabla^2 r_i. $$

残差小时，第二项可以忽略，只用 $J^\top J$ 就是高斯–牛顿法：不需要算二阶导数，且 $J^\top J$ 总是半正定（6.6 节）。例题：$f=x^2+3xy+y^2$，$H=\begin{bmatrix}2&3\\3&2\end{bmatrix}$，$\det H=4-9=-5<0$，原点是鞍点；特征值 $5,\,-1$，特征向量 $(1,1)$（向上弯）与 $(1,-1)$（向下弯）。

When the residuals are small the second term can be dropped, and using only $J^\top J$ is the Gauss–Newton method: no second derivatives needed, and $J^\top J$ is always positive semi-definite (Lesson 6.6). Example: $f=x^2+3xy+y^2$ has $H=\begin{bmatrix}2&3\\3&2\end{bmatrix}$ and $\det H=4-9=-5<0$, so the origin is a saddle; the eigenvalues are $5,\,-1$, with eigenvectors $(1,1)$ (bends up) and $(1,-1)$ (bends down).

6机器人中的它：定位的不确定性椭圆In robotics: the localization uncertainty ellipse

机器人（绿点）在平面里，根据到几个已知路标（红色方块）的距离估计自己的位置。距离测量带有高斯噪声 $\sigma$，最大似然估计就是最小化负对数似然

$$ \text{NLL}(\mathbf x) = \sum_i \frac{\big(\|\mathbf x-\mathbf l_i\|-\rho_i\big)^2}{2\sigma^2}. $$

最小值在机器人真实位置处，那里 $\text{NLL}=0$，Hessian 是 $\;H=\tfrac1{\sigma^2}\sum_i\mathbf n_i\mathbf n_i^\top$（$\mathbf n_i$ 是从路标 $i$ 指向机器人的单位向量，也就是残差的梯度；残差为 0，所以高斯–牛顿项就是全部）。它是估计的信息矩阵，逆矩阵 $H^{-1}$ 给出估计误差的协方差，紫色虚线椭圆就是 $\boldsymbol\delta^\top H\boldsymbol\delta=1$ 和 $=4$，即 1σ 与 2σ 的不确定性椭圆。

拖动路标和机器人。三个路标分散时椭圆接近圆；只有两个路标、并且机器人恰好和它们共线时，视线方向 $\mathbf n_1\parallel\mathbf n_2$，$H$ 只剩一个非零特征值，椭圆被拉成细长条：垂直于连线的方向无法确定。

A robot (green dot) in the plane estimates its position from distances to a few known landmarks (red squares). The range measurements carry Gaussian noise $\sigma$, and maximum likelihood minimizes the negative log-likelihood

$$ \text{NLL}(\mathbf x) = \sum_i \frac{\big(\|\mathbf x-\mathbf l_i\|-\rho_i\big)^2}{2\sigma^2}. $$

The minimum is at the robot's true position, where $\text{NLL}=0$ and the Hessian is $\;H=\tfrac1{\sigma^2}\sum_i\mathbf n_i\mathbf n_i^\top$ (with $\mathbf n_i$ the unit vector from landmark to robot, i.e. the residual's gradient; the residuals vanish, so the Gauss–Newton term is everything). This is the estimate's information matrix, and its inverse $H^{-1}$ gives the covariance of the estimation error: the purple dashed ellipses are $\boldsymbol\delta^\top H\boldsymbol\delta=1$ and $=4$, the 1σ and 2σ uncertainty ellipses.

Drag the landmarks and the robot. With three well-spread landmarks the ellipse is nearly a circle; with only two landmarks and the robot exactly collinear with them, the bearing directions $\mathbf n_1\parallel\mathbf n_2$, $H$ keeps a single non-zero eigenvalue and the ellipse stretches into a thin sliver: the direction perpendicular to the line cannot be determined.

FIG 5Hessian 即信息矩阵The Hessian as an information matrix

颜色与实线是真实的 NLL 等高线（取值 0.5、2、4.5…，即 $\tfrac12k^2$），紫色虚线是用 $H^{-1}$ 画出的 1σ、2σ 椭圆，青色细线是机器人到路标的视线。离机器人越远，二次近似与真实等高线的偏差越大。读数里的"数值 − 解析"是对 NLL 做数值二阶差分得到的 Hessian 与公式的最大差。 The shading and solid lines are the true NLL contours (levels 0.5, 2, 4.5, …, i.e. $\tfrac12k^2$); the purple dashed lines are the 1σ and 2σ ellipses drawn from $H^{-1}$; the thin cyan lines are the lines of sight from robot to landmarks. Farther from the robot the quadratic approximation drifts from the true contours. The readout "numeric − analytic" is the largest difference between a Hessian obtained by numerically differencing the NLL and the formula.

用 3 个路标，把椭圆调成接近圆（$\lambda_{\max}/\lambda_{\min}<1.5$）：路标从机器人看过去应该分布在哪些方向？With 3 landmarks, make the ellipse nearly circular ($\lambda_{\max}/\lambda_{\min}<1.5$): in which directions, seen from the robot, should the landmarks lie?
换成 2 个路标，让机器人与它们大致共线，把椭圆拉成细条（$\lambda_{\max}/\lambda_{\min}>30$）。Switch to 2 landmarks and put the robot roughly in line with them to stretch the ellipse into a sliver ($\lambda_{\max}/\lambda_{\min}>30$).

真实系统里In real systems

SLAM 与因子图：每个因子贡献一块 $J^\top\Sigma^{-1}J$，叠加起来就是整个问题的 Hessian（信息矩阵）；"稀疏"就体现在这个矩阵里（6.9 节）。卡尔曼滤波的信息形式直接累加信息矩阵（4.8 节）。主动感知：规划让信息矩阵的最小特征值变大的路径，即把最不确定的方向照亮。退化检测：LiDAR 里程计在长走廊里沿走廊方向的 Hessian 特征值接近 0，就可以判定该方向退化，不要相信那个方向上的估计。训练神经网络：Hessian 的特征值谱（尖锐程度）常被用来讨论极小点的平坦程度与泛化。

SLAM and factor graphs: each factor contributes a block $J^\top\Sigma^{-1}J$, and summing them gives the Hessian (information matrix) of the whole problem; the "sparsity" lives in this matrix (Lesson 6.9). The information form of the Kalman filter simply accumulates information matrices (Lesson 4.8). Active perception: plan paths that raise the information matrix's smallest eigenvalue, i.e. shine light on the most uncertain direction. Degeneracy detection: in a long corridor a LiDAR odometry Hessian has an eigenvalue near 0 along the corridor, flagging that direction as degenerate, so do not trust the estimate along it. Neural-network training: the Hessian's eigenvalue spectrum (sharpness) is often used to discuss the flatness of minima and generalization.

7常见误区Common pitfalls

误区 1：只看对角元素判断凸性Pitfall 1: judging convexity from the diagonal alone

$H=\begin{bmatrix}1&2\\2&1\end{bmatrix}$ 的对角元素都是正的，可是 $\det H=-3<0$，特征值是 3 和 $-1$，是鞍面（试着在 FIG 2 里设 $a=c=1,b=2$，看看红色虚线）。要看特征值（或用 Sylvester 判据：所有顺序主子式为正）。

$H=\begin{bmatrix}1&2\\2&1\end{bmatrix}$ has positive diagonal entries, yet $\det H=-3<0$ and the eigenvalues are 3 and $-1$: a saddle (set $a=c=1,b=2$ in FIG 2 and watch the red dashed lines). Look at the eigenvalues (or use Sylvester's criterion: all leading principal minors positive).

误区 2：牛顿步永远是下降方向Pitfall 2: the Newton step is always a descent direction

只有 $H\succ0$ 时 $-H^{-1}\nabla f$ 才保证是下降方向。非凸问题里，纯牛顿步可能飞向鞍点甚至极大点（FIG 4 的红色步）。实用的牛顿类方法都带阻尼、线搜索或信赖域。

$-H^{-1}\nabla f$ is guaranteed to be a descent direction only if $H\succ0$. In non-convex problems a pure Newton step can fly to a saddle or even a maximum (the red steps in FIG 4). Practical Newton-type methods always come with damping, a line search, or a trust region.

误区 3：二次近似"到处"成立Pitfall 3: the quadratic model holds everywhere

二阶泰勒的误差是 $O(\|\boldsymbol\delta\|^3)$，只在 $\mathbf p$ 附近可靠。FIG 3 里把半径放大，虚线与实线很快分道扬镳；FIG 5 里的 2σ 椭圆在远处也不再吻合真实等高线。用 Hessian 估计不确定性（拉普拉斯近似）时，别忘了它只在最优点附近成立。

The second-order Taylor error is $O(\|\boldsymbol\delta\|^3)$, reliable only near $\mathbf p$. Enlarge the radius in FIG 3 and the dashed contours soon part ways with the solid ones; in FIG 5 the 2σ ellipse also stops matching the true contours farther out. When using the Hessian to estimate uncertainty (the Laplace approximation), remember it holds only near the optimum.

误区 4：混淆雅可比与 Hessian 的形状Pitfall 4: mixing up the shapes of the Jacobian and the Hessian

标量函数 $f:\R^n\to\R$：梯度是 $n$ 维向量，Hessian 是 $n\times n$ 矩阵。向量函数 $F:\R^n\to\R^m$ 的"二阶导数"是 $m$ 个 Hessian（一个 $m\times n\times n$ 张量），不是一个矩阵。机器人里只对标量代价写"Hessian"。

For a scalar function $f:\R^n\to\R$, the gradient is an $n$-vector and the Hessian an $n\times n$ matrix. A vector function $F:\R^n\to\R^m$ has $m$ Hessians (an $m\times n\times n$ tensor) as its "second derivative", not one matrix. In robotics one writes "the Hessian" only for scalar costs.

误区 5：数值求二阶导数用很小的步长Pitfall 5: using a tiny step for numerical second derivatives

二阶差分 $(f(x+h)-2f(x)+f(x-h))/h^2$ 的舍入误差按 $\epsilon_{\text{mach}}/h^2$ 放大，比一阶差分（$\epsilon/h$）严重得多。双精度下合适的 $h$ 大约是 $10^{-4}$，而不是 $10^{-8}$；能解析求、或者用自动微分，就不要用差分。

The round-off of the second difference $(f(x+h)-2f(x)+f(x-h))/h^2$ grows like $\epsilon_{\text{mach}}/h^2$, much worse than for a first difference ($\epsilon/h$). In double precision a good $h$ is around $10^{-4}$, not $10^{-8}$; prefer analytic derivatives or automatic differentiation to finite differences.

8小练习Check yourself

1. $f(x,y)=x^2y$，它在点 $(1,2)$ 的 Hessian 是？1. What is the Hessian of $f(x,y)=x^2y$ at $(1,2)$?

$f_x=2xy$，$f_y=x^2$。$f_{xx}=2y=4$，$f_{xy}=2x=2$，$f_{yy}=0$。Hessian 必须对称，所以选项 C（非对称）一定错了。$f_x=2xy$, $f_y=x^2$. $f_{xx}=2y=4$, $f_{xy}=2x=2$, $f_{yy}=0$. A Hessian must be symmetric, so option C (asymmetric) is wrong at a glance.

2. $f(x,y)=x^2+3xy+y^2$ 在原点（梯度为 0）是什么点？2. What kind of point is the origin (zero gradient) of $f(x,y)=x^2+3xy+y^2$?

$H=\begin{bmatrix}2&3\\3&2\end{bmatrix}$，$\det H=4-9=-5<0$，特征值 5 与 $-1$。沿 $(1,1)$ 方向向上弯，沿 $(1,-1)$ 向下弯。交叉项 $3xy$ 足以让它变成鞍面，这正是误区 1。$H=\begin{bmatrix}2&3\\3&2\end{bmatrix}$, $\det H=4-9=-5<0$, eigenvalues 5 and $-1$: it bends up along $(1,1)$ and down along $(1,-1)$. The cross term $3xy$ is enough to make it a saddle, which is Pitfall 1.

3. 用固定学习率 $\eta$ 对 $f=\tfrac12(x^2+10y^2)$ 做梯度下降。要保证收敛，$\eta$ 必须小于？3. You run gradient descent with a fixed rate $\eta$ on $f=\tfrac12(x^2+10y^2)$. To guarantee convergence, $\eta$ must be below…?

$H=\operatorname{diag}(1,10)$，$\lambda_{\max}=10$，要求 $\eta<2/\lambda_{\max}=0.2$。受限于最陡的方向 $y$：$y\leftarrow(1-10\eta)y$。这一步长对平缓的 $x$ 方向却太小：每步只缩小 $1-\eta\ge0.8$，这就是条件数 $\kappa=10$ 的代价。$H=\operatorname{diag}(1,10)$, $\lambda_{\max}=10$, so we need $\eta<2/\lambda_{\max}=0.2$. The steepest direction $y$ is the limit: $y\leftarrow(1-10\eta)y$. Yet that step is too small for the gentle $x$ direction, which only shrinks by $1-\eta\ge0.8$ per step: the price of the condition number $\kappa=10$.

4. 对一个严格凸的二次函数 $f=\tfrac12\mathbf x^\top H\mathbf x-\mathbf b^\top\mathbf x$（$H\succ0$），从任意起点出发，纯牛顿法需要几步到达极小点？4. For a strictly convex quadratic $f=\tfrac12\mathbf x^\top H\mathbf x-\mathbf b^\top\mathbf x$ ($H\succ0$), how many pure Newton steps from any start reach the minimizer?

二次函数的二阶泰勒模型就是它自己，牛顿步 $-H^{-1}(H\mathbf x-\mathbf b)=H^{-1}\mathbf b-\mathbf x$，加到 $\mathbf x$ 上恰好得到 $H^{-1}\mathbf b$，也就是极小点。A quadratic's second-order Taylor model is the function itself. The Newton step is $-H^{-1}(H\mathbf x-\mathbf b)=H^{-1}\mathbf b-\mathbf x$, and adding it to $\mathbf x$ lands exactly on $H^{-1}\mathbf b$, the minimizer.

5. 定位问题在最优点处的 Hessian（信息矩阵）为 $H=\operatorname{diag}(4,1)$。估计误差的协方差约为 $H^{-1}$，哪个说法正确？5. A localization problem's Hessian (information matrix) at the optimum is $H=\operatorname{diag}(4,1)$. The estimation-error covariance is about $H^{-1}$. Which statement is right?

$H^{-1}=\operatorname{diag}(0.25,1)$，标准差是方差的平方根：0.5 与 1。曲率大（$H$ 大）意味着代价在该方向"陡"，数据对这个方向约束得紧，不确定性小。信息矩阵与协方差互为逆矩阵。$H^{-1}=\operatorname{diag}(0.25,1)$, and standard deviations are the square roots of the variances: 0.5 and 1. A large curvature means the cost is "steep" in that direction, the data constrain it tightly, and the uncertainty is small. The information matrix and the covariance are inverses of each other.

9延伸阅读Further reading

S. Boyd & L. Vandenberghe, Convex Optimization, Appendix A.4 & Ch. 9.5 — Hessian、凸性的二阶条件与牛顿法The Hessian, second-order convexity conditions and Newton's method
J. Nocedal & S. Wright, Numerical Optimization, Ch. 2 & 3 & 10 — 二阶最优性条件、牛顿法、高斯–牛顿与 Levenberg–MarquardtSecond-order optimality conditions, Newton's method, Gauss–Newton and Levenberg–Marquardt
Khan Academy, Multivariable Calculus — 二阶偏导数、二阶泰勒近似与二次型的视频Videos on second partials, quadratic approximation and quadratic forms
G. Strang, Linear Algebra and Its Applications — 正定矩阵与二次型（谱定理的应用）Positive definite matrices and quadratic forms (an application of the spectral theorem)
T. Barfoot, State Estimation for Robotics, Ch. 2 & 4 — 最大后验估计、高斯–牛顿与信息矩阵的几何意义Maximum a posteriori estimation, Gauss–Newton and the geometry of the information matrix

\(H\) 的特征值Eigenvalues of \(H\)	名称Name	驻点类型Stationary point
全为正all positive	正定 \(H\succ0\)positive definite \(H\succ0\)	严格局部极小strict local minimum
全为负all negative	负定 \(H\prec0\)negative definite \(H\prec0\)	严格局部极大strict local maximum
有正有负mixed signs	不定indefinite	鞍点saddle point
有 0（其余同号）some zero (rest same sign)	半定、奇异semi-definite, singular	不能下结论，需看更高阶inconclusive; need higher order