Part I · 第 4 章概率 · 4.6Part I · Ch. 4 Probability · 4.6

最大似然与最大后验Maximum Likelihood & MAP

手里有一把带噪声的读数，参数到底是多少？最大似然 (MLE) 的回答是：挑那个让"我真的看到这批数据"最不意外的值。噪声是高斯的，它就恰好等于最小二乘；噪声换成别的形状，就变成别的损失。如果你还有先验知识（地图、数据手册、上一次的标定），把它乘进去，就得到最大后验 (MAP)，它在公式里表现为"正则项"。 You hold a handful of noisy readings. What is the parameter? Maximum likelihood (MLE) answers: pick the value that makes the data you actually saw the least surprising. If the noise is Gaussian this is exactly least squares; change the noise shape and you get a different loss. If you also have prior knowledge (a map, a datasheet, last week's calibration), multiply it in and you get maximum a posteriori (MAP), which shows up in the formulas as a "regularizer".

约 20 分钟~20 min 前置：4.2 条件概率与贝叶斯 · 4.4 高斯分布 · 1.13 最小二乘Prereqs: 4.2 Conditional Probability & Bayes · 4.4 Gaussians · 1.13 Least Squares 4 个交互4 interactives

1为什么机器人需要它Why robots need this

机器人的世界里到处是"从带噪声的数据里反推一个数"：超声波传感器对着墙连测 20 次，墙到底有多远？轮子转了几千圈的日志，轮半径是多少？相机标定、陀螺仪零偏、电机常数、激光点云拟合出的墙面……每一个都是同一个问题：参数 $\theta$ 未知，手里有数据 $z_1,\dots,z_n$，怎么估？

最大似然给出一个有原则的统一回答，并解释了一件你可能一直当作"理所当然"的事：为什么大家默认用平方误差。因为那等价于假设噪声是高斯的。一旦传感器有野值（激光打到了路人的腿），高斯假设就错了，损失函数也该换。

再往前一步：机器人通常不是从零开始。地图说墙在 2.4 米外，数据手册说这个电机的转矩常数约 0.05。最大后验把这些先验和数据放在同一个公式里，数据少时靠先验兜底，数据多时让数据说话。这也是 4.7 节贝叶斯滤波与 4.8 节卡尔曼滤波的预演：滤波器做的，正是"每来一个读数就更新一次后验"。

Robotics is full of "recover one number from noisy data": an ultrasonic sensor pings a wall 20 times, so how far is the wall? A log of thousands of wheel revolutions gives the wheel radius. Camera calibration, gyroscope bias, motor constants, a wall fitted to a lidar scan: all the same question. The parameter $\theta$ is unknown, you hold data $z_1,\dots,z_n$; how do you estimate it?

Maximum likelihood gives one principled answer, and it explains something you may have taken for granted: why everybody defaults to squared error. It is equivalent to assuming the noise is Gaussian. As soon as the sensor produces outliers (a lidar beam hits a pedestrian's leg), the Gaussian assumption is wrong and the loss should change too.

One step further: a robot rarely starts from nothing. The map says the wall is 2.4 m away; the datasheet says the motor's torque constant is about 0.05. MAP puts the prior and the data in one formula: with little data it falls back on the prior, with lots of data the data speaks. This is also a rehearsal for Lesson 4.7 (Bayes filter) and 4.8 (Kalman filter): a filter does exactly "update the posterior every time one more reading arrives".

2直觉：固定数据，让参数滑动Intuition: freeze the data, slide the parameter

概率 $p(z\mid\theta)$ 通常的读法是"参数已知，数据会长什么样"。似然 (likelihood) 把它倒过来读：数据已经看到了、固定不动，把 $\theta$ 当作变量，问"哪个 $\theta$ 最能解释这批数据？"。

设每个读数是 $z_i = \theta + \text{噪声}$，噪声标准差 $\sigma$ 已知。给定一个候选 $\theta$，在它周围立一座高斯小山。每个数据点都"落在山的某个高度上"，这个高度就是 $p(z_i\mid\theta)$；把所有高度相乘（数据相互独立），就是这个 $\theta$ 的似然。拖动小山，看哪里的乘积最大。

A probability $p(z\mid\theta)$ is normally read as "the parameter is known; what will the data look like?". The likelihood reads it the other way round: the data are already seen and frozen; $\theta$ is the variable, and the question is "which $\theta$ best explains this data?".

Say each reading is $z_i = \theta + \text{noise}$ with known noise standard deviation $\sigma$. For a candidate $\theta$, erect a Gaussian hill around it. Every data point sits "at some height on the hill"; that height is $p(z_i\mid\theta)$. Multiply all the heights (the data are independent) and you have this $\theta$'s likelihood. Drag the hill and see where the product peaks.

FIG 1似然地形：滑动 θ，找乘积最大处The likelihood landscape: slide θ, find the biggest product

数据与候选高斯 N(θ, σ²)DATA AND CANDIDATE GAUSSIAN N(θ, σ²)

θ 的似然（相对峰值）LIKELIHOOD OF θ (RELATIVE TO ITS PEAK)

紫点是读数（可以左右拖动），青色小山是候选的 N(θ, σ²)，拖它的山顶或下方曲线上的圆圈改变 θ。每个紫点上方的竖线高度就是 $p(z_i\mid\theta)$。下图把"所有竖线高度的乘积"画成 θ 的函数；黄色竖线是它的最大值点，恰好是样本均值。右上的开关可以改看对数似然，乘积变成了求和。 Purple dots are the readings (drag them sideways); the cyan hill is the candidate N(θ, σ²), and you move θ by dragging its peak or the circle on the lower curve. The height of the stem above each purple dot is $p(z_i\mid\theta)$. The lower plot draws "the product of all stem heights" as a function of θ; the amber line marks its maximum, which is exactly the sample mean. Use the switch to view the log-likelihood, where the product becomes a sum.

试一试Try this

把 θ 拖到似然的最高点（误差小于 0.05）。它和样本均值是什么关系？Drag θ onto the top of the likelihood (within 0.05). How does it relate to the sample mean?
点"加一个读数"，直到 n ≥ 10。似然的山峰是变宽还是变窄？宽度约为 σ/√n。Press "Add a reading" until n ≥ 10. Does the likelihood peak widen or narrow? Its width is about σ/√n.
把一个紫点拖到最右边，使最大似然估计超过 4.1（原来是 3.4）。一个野值能把均值拉多远？Drag one purple dot far right so the ML estimate exceeds 4.1 (it starts at 3.4). How far can one outlier pull the mean?

核心洞见Key insight

似然是 $\theta$ 的函数，不是 $\theta$ 的概率分布：它在 $\theta$ 上的面积不必等于 1。它只告诉你"相对来说哪个 $\theta$ 更能解释数据"。把小山拖到别处你会看到：所有竖线几乎同时变矮，乘积以指数速度掉下去，所以实际计算总是用对数似然，把乘积变成求和。

The likelihood is a function of $\theta$, not a probability distribution over $\theta$: its area under $\theta$ need not be 1. It only says which $\theta$ explains the data relatively better. Drag the hill away and every stem shrinks at once and the product drops exponentially, which is why practice always works with the log-likelihood, turning the product into a sum.

3动手玩：噪声假设决定损失函数Play: the noise assumption picks the loss

把 FIG 1 的高斯小山取对数：$\log p(z_i\mid\theta) = -\frac{(z_i-\theta)^2}{2\sigma^2} + \text{常数}$。最大化它们的和，等价于最小化平方误差之和。你在 1.13 节里学的最小二乘，原来就是"假设噪声是高斯的"的最大似然。

但这是一个假设。如果噪声更容易出现大偏差（拉普拉斯分布、柯西分布等"重尾"分布），负对数似然就不再是二次的：拉普拉斯给出绝对值误差，柯西给出 $\log(1+r^2)$，它们对远离的点的惩罚增长得慢得多。下面的实验室让你直接对比：拖动数据点，看三种噪声假设的拟合线怎么分道扬镳。

Take the log of FIG 1's Gaussian hill: $\log p(z_i\mid\theta) = -\frac{(z_i-\theta)^2}{2\sigma^2} + \text{const}$. Maximizing their sum is the same as minimizing the sum of squared errors. The least squares of Lesson 1.13 is, it turns out, maximum likelihood under the assumption that the noise is Gaussian.

But that is an assumption. If the noise produces big deviations more often (Laplace, Cauchy, other "heavy-tailed" distributions), the negative log-likelihood is no longer quadratic: Laplace gives the absolute error, Cauchy gives $\log(1+r^2)$, and both penalize far-away points much less. The lab below lets you compare directly: drag the data points and watch the fits under three noise assumptions part ways.

FIG 2直线拟合：高斯、拉普拉斯与柯西噪声Line fitting under Gaussian, Laplace and Cauchy noise

七个数据点可以任意拖动（最右边那个是故意放的野值）。三条线分别最小化 $\sum r_i^2$（红，高斯）、$\sum |r_i|$（绿，拉普拉斯）、$\sum \log(1+r_i^2)$（蓝，柯西）；被选中的那条画粗，并画出各点到它的残差。右上角的小图是三种损失 ρ(r)：红色抛物线增长最快，所以野值对它的"拉力"最大。 The seven points can be dragged anywhere (the rightmost one is a deliberate outlier). The three lines minimize $\sum r_i^2$ (red, Gaussian), $\sum |r_i|$ (green, Laplace) and $\sum \log(1+r_i^2)$ (blue, Cauchy); the selected one is drawn thick, with residuals to it. The inset at the top right shows the three losses ρ(r): the red parabola grows fastest, so an outlier pulls on it hardest.

试一试Try this

切换到拉普拉斯或柯西：让它和高斯拟合在 x = 9 处相差超过 0.8，也就是亲眼看到野值把红线拉歪。Switch to Laplace or Cauchy and make it differ from the Gaussian fit by more than 0.8 at x = 9, i.e. see the outlier bend the red line with your own eyes.
把野值拖回直线附近：三条线在 x = 5 和 x = 9 处都相差不到 0.15。数据干净时，噪声假设几乎无所谓。Drag the outlier back near the line until the three fits differ by less than 0.15 at both x = 5 and x = 9. With clean data the noise assumption hardly matters.

和 1.13 节的关系Link to Lesson 1.13

把读数堆成向量，$\mathbf z = A\boldsymbol\theta + \boldsymbol\epsilon$，$\boldsymbol\epsilon\sim\mathcal N(\mathbf 0,\sigma^2 I)$。对数似然就是 $-\|\mathbf z - A\boldsymbol\theta\|^2/(2\sigma^2)$ 加常数，它的最大值点就是 $(A^\top A)^{-1}A^\top\mathbf z$，即正规方程的解。噪声方差各不相同时，得到 1.13 节的加权最小二乘，权重就是方差的倒数。红线是这种解；绿线、蓝线没有闭式解，通常用"迭代重加权最小二乘"（每轮给残差大的点降权）求，这里的实现就是如此。柯西损失不是凸的，所以这里从绿线出发迭代，一般的做法也是如此。

Stack the readings into a vector, $\mathbf z = A\boldsymbol\theta + \boldsymbol\epsilon$, $\boldsymbol\epsilon\sim\mathcal N(\mathbf 0,\sigma^2 I)$. The log-likelihood is $-\|\mathbf z - A\boldsymbol\theta\|^2/(2\sigma^2)$ plus a constant, and its maximizer is $(A^\top A)^{-1}A^\top\mathbf z$, the solution of the normal equations. With different noise variances per reading you get Lesson 1.13's weighted least squares, with weights equal to inverse variances. The red line is that solution; the green and blue lines have no closed form and are usually found by "iteratively reweighted least squares" (down-weight points with large residuals each round), which is what this page does. The Cauchy loss is not convex, so we start it from the green fit, as is common practice.

4第二个视角：加上先验就是 MAPA second view: add a prior and you get MAP

贝叶斯定理（4.2 节）说 $p(\theta\mid z)\propto p(z\mid\theta)\,p(\theta)$：后验 = 似然 × 先验。最大后验 (MAP) 就是取后验的最高点。MLE 只看数据；MAP 还多一个乘法因子，把"在看数据之前我对 $\theta$ 的看法"也算进去。

最干净的例子是标量高斯：先验 $\theta\sim\mathcal N(\mu_0,\tau^2)$，$n$ 个读数，噪声 $\sigma$。三条曲线都是高斯，后验的均值是先验均值与数据均值的按精度加权平均。FIG 3 里每一个量都能拖、能调。

Bayes' theorem (Lesson 4.2) says $p(\theta\mid z)\propto p(z\mid\theta)\,p(\theta)$: posterior = likelihood × prior. Maximum a posteriori (MAP) takes the top of the posterior. MLE looks only at the data; MAP has one extra factor that brings in "what I believed about $\theta$ before looking".

The cleanest example is a scalar Gaussian: prior $\theta\sim\mathcal N(\mu_0,\tau^2)$, $n$ readings with noise $\sigma$. All three curves are Gaussians, and the posterior mean is a precision-weighted average of the prior mean and the data mean. Every quantity in FIG 3 can be dragged or tuned.

FIG 3先验、似然与后验：MAP 向先验"收缩"Prior, likelihood, posterior: MAP "shrinks" toward the prior

青色 = 先验 N(μ₀, τ²)，紫色 = 数据给出的似然 N(z̄, σ²/n)，黄色 = 后验。竖线：蓝 = ML，黄 = MAP，灰色虚线 = 真值（滤波器并不知道）。拖先验底部的圆点移动 μ₀，拖它腰部的圆点改变 τ。底部的小刻度是已用到的读数。 Cyan = prior N(μ₀, τ²), purple = the likelihood from the data, N(z̄, σ²/n), amber = posterior. Vertical lines: blue = ML, amber = MAP, dashed grey = the true value (which the estimator does not know). Drag the dot at the base of the prior to move μ₀, the dot on its flank to change τ. The small ticks at the bottom are the readings used so far.

试一试Try this

数据少、先验强：n 在 1 到 3 之间，让数据所占权重 w 低于 0.3。MAP 离 ML 比离先验远还是近？Little data, strong prior: with n between 1 and 3, push the data weight w below 0.3. Is MAP closer to the prior or to the ML estimate?
先验错了，但数据够多：让 μ₀ 离数据均值至少 2，同时 w 超过 0.95。先验几乎被淹没。Wrong prior, plenty of data: put μ₀ at least 2 away from the data mean while w exceeds 0.95. The prior is nearly drowned out.

先验 = 正则项A prior is a regularizer

取负对数，MAP 就是最小化 $\dfrac{1}{2\sigma^2}\sum_i (z_i-\theta)^2 + \dfrac{1}{2\tau^2}(\theta-\mu_0)^2$：数据拟合项 + 惩罚偏离先验的项。两边乘以 $2\sigma^2$，惩罚系数就是 $\lambda^2 = \sigma^2/\tau^2$。这正是 1.13 节的阻尼最小二乘 / 岭回归（取 $\mu_0=0$）：阻尼 λ 越大，等价于"我越相信参数应当很小"。选 λ，就是在说你对先验的信心。

Take negative logs and MAP minimizes $\dfrac{1}{2\sigma^2}\sum_i (z_i-\theta)^2 + \dfrac{1}{2\tau^2}(\theta-\mu_0)^2$: a data-fit term plus a penalty for straying from the prior. Multiply by $2\sigma^2$ and the penalty weight is $\lambda^2 = \sigma^2/\tau^2$. This is exactly Lesson 1.13's damped least squares / ridge regression (with $\mu_0=0$): a larger damping λ means "I strongly believe the parameters should be small". Choosing λ is stating how much you trust the prior.

5再看数学The math

提示：悬停下面公式中带底色的符号，FIG 3 中对应的曲线或竖线会高亮。Tip: hover the shaded symbols below and the matching curve or line in FIG 3 lights up.

最大似然Maximum likelihood

读数相互独立，似然是各个概率之积；对数不改变最大值点，却把积变成和：

The readings are independent, so the likelihood is the product of their probabilities; the log keeps the maximizer but turns the product into a sum:

$$ \hat\theta_{\mathrm{ML}} = \arg\max_\theta \hl{lik}{\cw{\prod_{i=1}^n p(z_i\mid\theta)}} = \arg\max_\theta \sum_{i=1}^n \log p(z_i\mid\theta) = \arg\min_\theta \underbrace{-\sum_{i=1}^n \log p(z_i\mid\theta)}_{\text{negative log-likelihood}} . $$

噪声是高斯 $\mathcal N(0,\sigma^2)$ 时，$-\log p(z_i\mid\theta) = \frac{(z_i-\theta)^2}{2\sigma^2} + \log(\sigma\sqrt{2\pi})$。于是

With Gaussian noise $\mathcal N(0,\sigma^2)$, $-\log p(z_i\mid\theta) = \frac{(z_i-\theta)^2}{2\sigma^2} + \log(\sigma\sqrt{2\pi})$. Therefore

$$ \hat\theta_{\mathrm{ML}} = \arg\min_\theta \sum_i (z_i-\theta)^2 \;\Longrightarrow\; \frac{d}{d\theta}:\; -2\sum_i (z_i-\hat\theta)=0 \;\Longrightarrow\; \hl{ml}{\ck{\hat\theta_{\mathrm{ML}}}} = \bar z = \frac1n\sum_i z_i . $$

换成拉普拉斯噪声（密度 $\propto e^{-|r|/b}$），负对数似然是 $\sum|z_i-\theta|/b$，最大似然估计变成中位数。这就是为什么中位数比均值对野值更"皮实"。对于 $\sigma$ 本身，最大似然给出 $\hat\sigma^2_{\mathrm{ML}} = \frac1n\sum_i (z_i-\bar z)^2$（分母是 $n$）。它的期望是 $\frac{n-1}{n}\sigma^2$，系统性偏小，所以常把分母改成 $n-1$。

With Laplace noise (density $\propto e^{-|r|/b}$) the negative log-likelihood is $\sum|z_i-\theta|/b$, and the ML estimate becomes the median. That is why the median is sturdier against outliers than the mean. For $\sigma$ itself, ML gives $\hat\sigma^2_{\mathrm{ML}} = \frac1n\sum_i (z_i-\bar z)^2$ (denominator $n$). Its expectation is $\frac{n-1}{n}\sigma^2$, systematically too small, which is why the denominator is often changed to $n-1$.

最大后验Maximum a posteriori

对贝叶斯公式取对数。分母 $p(z)$ 与 $\theta$ 无关，在 argmax 中消失：

Take the log of Bayes' rule. The denominator $p(z)$ does not depend on $\theta$ and vanishes inside the argmax:

$$ \hat\theta_{\mathrm{MAP}} = \arg\max_\theta \Big[\underbrace{\sum_i \log p(z_i\mid\theta)}_{\text{data}} + \hl{prior}{\cx{\log p(\theta)}}\Big] = \arg\min_\theta \Big[\sum_i \frac{(z_i-\theta)^2}{2\sigma^2} + \frac{(\theta-\mu_0)^2}{2\tau^2}\Big]. $$

令导数为零，整理成精度（方差的倒数）加权的形式，其中 $\bar z$ 是数据均值：

Setting the derivative to zero and rearranging gives the precision-weighted form, with $\bar z$ the data mean:

$$ \frac{1}{\sigma_{\mathrm{post}}^2} = \frac{1}{\cx{\tau^2}} + \frac{n}{\cw{\sigma^2}},\qquad \hl{map}{\cv{\hat\theta_{\mathrm{MAP}}}} = \hl{post}{\sigma_{\mathrm{post}}^2}\Big(\frac{\mu_0}{\tau^2} + \frac{n\bar z}{\sigma^2}\Big) = w\,\hl{ml}{\ck{\bar z}} + (1-w)\,\cx{\mu_0},\quad w = \frac{n/\sigma^2}{n/\sigma^2 + 1/\tau^2}. $$

这和 4.8 节两个高斯相乘的公式一模一样（那里的预测就是先验，测量就是似然）。三个极端情形值得记住：$n=0$ 时 $w=0$，MAP 就是先验；$n\to\infty$ 时 $w\to1$，MAP 趋近 MLE；$\tau\to\infty$（先验无信息）时同样 $w\to 1$，MAP = MLE。

This is the same formula as multiplying two Gaussians in Lesson 4.8 (there the prediction is the prior and the measurement is the likelihood). Three extremes are worth remembering: $n=0$ gives $w=0$ and MAP is the prior; $n\to\infty$ gives $w\to1$ and MAP approaches MLE; $\tau\to\infty$ (an uninformative prior) also gives $w\to 1$, so MAP = MLE.

先验的形状 = 正则项的形状The shape of the prior = the shape of the regularizer

先验 $p(\theta)$Prior $p(\theta)$	-log 先验−log prior	名字Name
高斯 $\mathcal N(0,\tau^2 I)$Gaussian $\mathcal N(0,\tau^2 I)$	$\\|\boldsymbol\theta\\|_2^2/(2\tau^2)$	岭回归 / 阻尼最小二乘（1.13 节）ridge / damped least squares (Lesson 1.13)
拉普拉斯Laplace	$\\|\boldsymbol\theta\\|_1/b$	Lasso，倾向于稀疏解lasso, favours sparse solutions
均匀（平坦）Uniform (flat)	常数constant	MAP = MLEMAP = MLE

机器学习里的"权重衰减 (weight decay)"，就是对网络权重施加了一个高斯先验。

"Weight decay" in machine learning is a Gaussian prior placed on the network's weights.

6机器人中的它：用地图先验估计到墙的距离In robotics: estimating the distance to a wall with a map prior

一台机器人面对一堵墙，用超声波测距，每次读数噪声标准差约 0.35 m。真实距离是 2.00 m。机器人手里有一张过时的地图，上面写着 2.4 m。机器人每测一次，就把"先验 + 目前所有读数"的后验算一次。

注意读数是一个一个到来的：每次的后验就是下一次的先验。这就是递推，也是贝叶斯滤波的雏形。拖动先验的均值与宽度，看 MAP 追上真值要多久。

A robot faces a wall and measures it with an ultrasonic sensor whose readings have a standard deviation of about 0.35 m. The true distance is 2.00 m. The robot owns an outdated map that says 2.4 m. After every reading it recomputes the posterior from "prior + all readings so far".

The readings arrive one at a time, and each posterior serves as the next prior. That is a recursion, and the seed of the Bayes filter. Drag the prior's mean and width and see how long MAP takes to catch up with the truth.

FIG 4逐个读数更新：ML 与 MAP 的竞赛One reading at a time: ML versus MAP

紫点是超声读数；蓝线是 ML（读数的运行平均）；黄线与黄带是 MAP 及其 ±2σ_post；青色虚线是地图先验；灰色虚线是真值 2.00 m。同一组读数，只是先验不同，估计就不同。 Purple dots are ultrasonic readings; the blue line is ML (the running average); the amber line and band are MAP with ±2σ_post; the cyan dashed line is the map prior; grey dashed is the truth, 2.00 m. Same readings; only the prior differs, and so does the estimate.

试一试Try this

先验错了 0.4 m，但读数足够多：n ≥ 30，MAP 与 ML 相差不到 5 cm。The prior is 0.4 m off, but with enough readings (n ≥ 30) MAP and ML differ by less than 5 cm.
把先验调得过度自信（τ ≤ 0.1），n ≥ 20 时 MAP 仍比 ML 偏离超过 10 cm。盲目相信一张旧地图的代价。Make the prior overconfident (τ ≤ 0.1): at n ≥ 20 MAP is still more than 10 cm away from ML. That is the price of blindly trusting an old map.

真实系统里In real systems

机器人里大量"拟合"其实是 MAP：SLAM 的位姿图优化最小化的就是"里程计约束残差 + 回环约束残差 + 先验"的加权平方和，权重来自各传感器的协方差（信息矩阵）；手眼标定、相机内参标定也是最大似然（重投影误差，噪声假设为高斯）；怀疑有野值时，换成 Huber 或柯西这样的鲁棒核函数，或者先用 RANSAC 剔除。

Much "fitting" in robotics is really MAP: SLAM pose-graph optimization minimizes a weighted sum of squared odometry residuals, loop-closure residuals and priors, with weights from each sensor's covariance (information matrix); hand-eye and camera-intrinsics calibration are maximum likelihood (reprojection error, Gaussian noise assumed); when outliers are suspected one swaps in a robust kernel such as Huber or Cauchy, or removes them first with RANSAC.

7常见误区Common pitfalls

误区 1：把似然当成 θ 的概率Pitfall 1: reading the likelihood as a probability of θ

$p(z\mid\theta)$ 对 $z$ 积分为 1，对 $\theta$ 积分不一定。"最大似然"是在选使数据最不意外的参数，并不是说"这个参数最可能为真"。要谈"θ 为真的概率"，必须有先验，得到后验才行。

$p(z\mid\theta)$ integrates to 1 over $z$, not necessarily over $\theta$. "Maximum likelihood" picks the parameter that makes the data least surprising; it does not say "this parameter is most probably true". To talk about the probability that θ is true you need a prior, and hence a posterior.

误区 2：数据很少时相信 MLEPitfall 2: trusting MLE with very little data

只有一个读数时，MLE 说"均值就是这个读数、方差是 0"：一个再自信不过的错误。MLE 会在小样本上过拟合。滤波器若用 MLE 估计自己的噪声方差，一旦 $\hat\sigma\to0$，增益就会变成"完全相信测量"或"完全相信模型"，从而发散。小样本时，更多先验或更多数据才是解药。

With a single reading, MLE says "the mean is this reading and the variance is 0": a supremely confident mistake. MLE overfits on small samples. If a filter estimated its own noise variance by MLE and $\hat\sigma\to0$, its gain would swing to "trust the measurement completely" or "trust the model completely" and it could diverge. With few samples, the cure is a prior or more data.

误区 3：以为 MAP 是"更好的 MLE"Pitfall 3: thinking MAP is "a better MLE"

先验错了，MAP 就带着偏差（FIG 4 里 τ 很小时就是如此）。而且后验的众数不随参数变换而协变：对 $\sigma$ 做 MAP 与对 $\sigma^2$ 做 MAP，得到的不是同一个点，因为概率密度在换元时要乘雅可比。"均匀先验"也依赖参数化：对 $\sigma$ 均匀，对 $\sigma^2$ 就不均匀。此外，MAP 只是后验的一个点，丢掉了后验的宽度。要保留不确定性，就保留整个后验，这正是贝叶斯滤波做的事。

If the prior is wrong, MAP carries a bias (as in FIG 4 with a small τ). Moreover the mode of a posterior does not transform covariantly: MAP for $\sigma$ and MAP for $\sigma^2$ are not the same point, because densities pick up a Jacobian under a change of variable. A "uniform prior" also depends on the parametrization: uniform in $\sigma$ is not uniform in $\sigma^2$. And MAP is a single point of the posterior; it throws away the posterior's width. To keep uncertainty, keep the whole posterior, which is what the Bayes filter does.

误区 4：默认用平方误差Pitfall 4: defaulting to squared error

平方误差意味着"相信噪声是高斯的"。激光打到玻璃、特征点误匹配、GPS 多路径，产生的是野值，FIG 2 里一个野值就能把红线拉歪。真实系统要么用鲁棒核（Huber、Cauchy、Geman–McClure），要么先做一致性检验（RANSAC、卡方门限）。

Squared error means "I believe the noise is Gaussian". A lidar hitting glass, a mismatched feature or GPS multipath produce outliers, and in FIG 2 a single one bends the red line. Real systems use robust kernels (Huber, Cauchy, Geman–McClure) or screen the data first (RANSAC, chi-square gating).

误区 5：把同一批数据既用来造先验、又用来算似然Pitfall 5: using the same data for the prior and the likelihood

先验应当来自"数据之外"的知识。若把上一轮估计结果当作这一轮的先验，却又重新使用了已经被用过的读数，信息就被重复计算，后验会过度自信。滤波器里的相关噪声（比如陀螺仪零偏）就容易造成这种事。

A prior should come from knowledge outside the data. If last round's estimate becomes this round's prior, and you then reuse readings that were already consumed, the information is counted twice and the posterior becomes overconfident. Correlated noise in a filter (a gyroscope's bias, say) invites exactly this.

8小练习Check yourself

1. 三个读数 1、2、6，噪声为高斯。θ 的最大似然估计是多少？如果噪声是拉普拉斯分布，又是多少？1. Three readings 1, 2, 6 with Gaussian noise. What is the ML estimate of θ? And if the noise were Laplace?

高斯噪声对应平方误差，最小值点是均值 $(1+2+6)/3=3$；拉普拉斯对应绝对值误差，最小值点是中位数 2。野值 6 把均值拉高，却几乎动不了中位数。Gaussian noise means squared error, minimized at the mean $(1+2+6)/3=3$; Laplace noise means absolute error, minimized at the median, 2. The outlier 6 drags the mean up and barely moves the median.

2. 先验 $\theta\sim\mathcal N(0,\,3)$（方差 $\tau^2=3$），噪声方差 $\sigma^2=1$，只有一个读数 $z=4$。MAP 估计是多少？2. Prior $\theta\sim\mathcal N(0,\,3)$ (variance $\tau^2=3$), noise variance $\sigma^2=1$, a single reading $z=4$. What is the MAP estimate?

$w = \dfrac{n/\sigma^2}{n/\sigma^2+1/\tau^2} = \dfrac{1}{1+1/3} = 0.75$，$\hat\theta_{\mathrm{MAP}} = 0.75\cdot 4 + 0.25\cdot 0 = 3$。读数比先验更可信（方差 1 对 3），所以结果偏向读数，但仍被先验往 0 拉了一点。$w = \dfrac{n/\sigma^2}{n/\sigma^2+1/\tau^2} = \dfrac{1}{1+1/3} = 0.75$, so $\hat\theta_{\mathrm{MAP}} = 0.75\cdot 4 + 0.25\cdot 0 = 3$. The reading is more trustworthy than the prior (variance 1 vs 3), so the answer leans toward it, but is still pulled a little toward 0.

3. 噪声标准差 $\sigma=2$，先验标准差 $\tau=1$（均值 0）。对应的岭回归 / 阻尼最小二乘中，$\lambda^2$ 是多少？3. Noise standard deviation $\sigma=2$, prior standard deviation $\tau=1$ (mean 0). In the corresponding ridge / damped least squares, what is $\lambda^2$?

$\lambda^2=\sigma^2/\tau^2 = 4/1 = 4$。先验越窄（τ 越小）或噪声越大（σ 越大），正则越强：你更相信"参数应该小"，而不是这批嘈杂的数据。$\lambda^2=\sigma^2/\tau^2 = 4/1 = 4$. A narrower prior (smaller τ) or noisier data (larger σ) means stronger regularization: you trust "the parameters should be small" more than this noisy data.

4. 只有一个读数 $z_1$，用最大似然（分母为 n）估计噪声方差 $\sigma^2$，得到多少？这说明了什么？4. With a single reading $z_1$, what does maximum likelihood (denominator n) give for the noise variance $\sigma^2$, and what does that tell you?

此时 $\bar z=z_1$，残差为 0，$\hat\sigma^2_{\mathrm{ML}}=\frac11\cdot 0=0$。一个读数无法告诉你噪声有多大，MLE 却断言"没有噪声"。用 $n-1$ 作分母得到 0/0（无定义），反而更诚实。Now $\bar z=z_1$, the residual is 0, and $\hat\sigma^2_{\mathrm{ML}}=\frac11\cdot 0=0$. One reading cannot tell you how noisy the sensor is, yet MLE asserts "no noise at all". With denominator $n-1$ you get 0/0 (undefined), which is more honest.

9延伸阅读Further reading

C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006, §1.2.5 & §3.3 — 最大似然、曲线拟合与"权重衰减 = 高斯先验"，以及贝叶斯线性回归maximum likelihood, curve fitting, "weight decay = Gaussian prior", and Bayesian linear regression
T. D. Barfoot, State Estimation for Robotics, Cambridge University Press, 2017, Ch. 2–3 — 机器人视角下的概率、MAP 与线性高斯估计probability, MAP and linear-Gaussian estimation from a robotics point of view
S. Boyd & L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004, §7.1 — 统计估计：ML 与 MAP 写成凸优化问题（线性测量模型下）statistical estimation: ML and MAP as convex optimization problems (for linear measurement models)
D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003, Ch. 2 & 3 — 似然原理与推断的直觉，免费电子版likelihood and inference with great intuition; free online edition
S. Thrun, W. Burgard & D. Fox, Probabilistic Robotics, MIT Press, 2005, Ch. 2 — 贝叶斯估计的基本概念与递推，4.7 节会用到the basic ideas of Bayesian estimation and its recursion, needed for Lesson 4.7

先验 \(p(\theta)\)Prior \(p(\theta)\)	-log 先验−log prior	名字Name
高斯 \(\mathcal N(0,\tau^2 I)\)Gaussian \(\mathcal N(0,\tau^2 I)\)	\(\\|\boldsymbol\theta\\|_2^2/(2\tau^2)\)	岭回归 / 阻尼最小二乘（1.13 节）ridge / damped least squares (Lesson 1.13)
拉普拉斯Laplace	\(\\|\boldsymbol\theta\\|_1/b\)	Lasso，倾向于稀疏解lasso, favours sparse solutions
均匀（平坦）Uniform (flat)	常数constant	MAP = MLEMAP = MLE