损失、泛化与正则化Losses, Generalization & Regularization

机器人越来越多的部件是从数据里学出来的：关节摩擦的补偿模型、“这次抓取会不会成功”的预测器、从人类示教学来的整套策略。16.1 节用最大似然拟合了回归和分类模型，并在留出的测试集上打分。训练时我们最小化的是训练集上的平均损失，可我们真正在乎的是机器人明天遇到的情况：新的物体、新的光照、新的速度。两者之差就是泛化问题。

有三件事决定泛化好不好。第一，损失函数：平方误差、L1、Huber 和交叉熵各自隐含着对噪声的不同假设，遇到传感器野值或类别标签时表现完全不同。第二，容量与正则化：一个参数比数据点多得多的网络能把训练误差压到几乎为零，包括把噪声也背下来；验证集、早停、L2、dropout、数据增广是防止这件事的工具。第三，对机器人最要命的是分布偏移：机器人自己的动作会改变它接下来看到的东西，部署时的输入分布和训练时不同，而模型在训练分布之外几乎没有任何保证（22.1 节的模仿学习会把这个问题放大成误差累积）。

More and more of a robot is learned from data: a compensation model for joint friction, a predictor of "will this grasp succeed", a whole policy learned from human demonstrations. Lesson 16.1 fitted regression and classification models by maximum likelihood and scored them on a held-out test set. During training we minimize the average loss on the training set, but what we really care about is what the robot meets tomorrow: new objects, new lighting, new speeds. The difference between the two is the generalization problem.

Three things decide how well a model generalizes. First, the loss: squared error, L1, Huber and cross-entropy each carry a different assumption about the noise, and they behave completely differently on sensor outliers or class labels. Second, capacity and regularization: a network with far more parameters than data points can drive its training error to almost zero, memorizing the noise along the way; validation sets, early stopping, L2, dropout and data augmentation are the tools against that. Third, and worst for robots, distribution shift: a robot's own actions change what it sees next, so the inputs at deployment are distributed differently from those in training, and a model has almost no guarantee outside its training distribution (imitation learning in Lesson 22.1 turns this into compounding errors).

4.6 节已经推出：如果测量噪声是高斯的，最大似然就是最小化平方误差；如果是拉普拉斯的（尾巴更厚），就是最小化绝对误差，估计值从均值变成中位数。换句话说，选损失就是选噪声模型。对拟合过程更直观的看法是每个数据点对拟合施加的“拉力”：损失 \(\hl{rho}{\rho(r)}\) 对残差 \(r\) 的导数 \(\hl{psi}{\psi(r)=\rho'(r)}\) 叫影响函数。平方损失的拉力与残差成正比，一个离谱的点拉力最大；L1 的拉力恒为 ±1，与残差大小无关；Huber 损失在 \(|r|\le\delta\) 时是平方、之外是线性，拉力被截在 ±δ。

下图是一个力传感器的标定：横轴是加的载荷，纵轴是读数，10 个点基本在一条直线上，外加一个可以拖动的点。把它拖成一个“野值”（比如接触瞬间的冲击读数），看三条拟合直线各被拉走多少。

Lesson 4.6 derived that if the measurement noise is Gaussian, maximum likelihood means minimizing the squared error; if it is Laplace (heavier tails), it means minimizing the absolute error, and the estimate moves from the mean to the median. In other words, choosing a loss is choosing a noise model. A more tangible view of what happens during fitting is the "pull" each data point exerts on the fit: the derivative \(\hl{psi}{\psi(r)=\rho'(r)}\) of the loss \(\hl{rho}{\rho(r)}\) with respect to the residual \(r\), called the influence function. Under the squared loss the pull grows with the residual, so the most absurd point pulls hardest; under L1 the pull is always ±1, whatever the residual; the Huber loss is quadratic for \(|r|\le\delta\) and linear beyond, so its pull is capped at ±δ.

The figure below is the calibration of a force sensor: load applied on the horizontal axis, reading on the vertical axis; 10 points lie roughly on a line, plus one point you can drag. Turn it into an "outlier" (say an impact spike at the moment of contact) and watch how far each of the three fitted lines gets pulled.

在默认位置 \(x=0.7\)，把这个点拖到离干净直线 1.5 V 时，平方拟合偏离 0.18 V，Huber（δ = 0.2）只偏离 0.03 V，L1 偏离 0.04 V；再往下拖，平方拟合继续跟着走（2.0 V 时 0.24 V），而 Huber 和 L1 几乎不再动：它们对这个点的拉力已经封顶了。代价是什么？把 δ 拉到 2，Huber 在这个范围内就退化成平方损失；而即使琥珀色圈就在直线上，L1 也和干净拟合差 0.02 V，因为它估计的是条件中位数而不是均值，在噪声真是高斯时浪费了一些数据。Huber 的阈值通常取噪声标准差的 1 到 2 倍左右（经典取法是 1.345σ，在高斯噪声下保留 95% 的效率），这里噪声 σ = 0.07 V，默认 δ = 0.2 V 约为 3σ。

At the default position \(x=0.7\), with the point dragged 1.5 V away from the clean line, the squared fit is off by 0.18 V while Huber (δ = 0.2) is off by only 0.03 V and L1 by 0.04 V; drag further and the squared fit keeps following (0.24 V at 2.0 V) while Huber and L1 hardly move: their pull from this point is already capped. What is the price? Raise δ to 2 and Huber turns back into the squared loss over this range; and L1 is 0.02 V off the clean fit even when the ring sits on the line, because it estimates the conditional median rather than the mean and wastes some data when the noise really is Gaussian. The Huber threshold is usually set at about 1 to 2 noise standard deviations (the classic choice 1.345σ keeps 95% efficiency under Gaussian noise); here the noise is σ = 0.07 V and the default δ = 0.2 V is about 3σ.

分类器输出一个 logit \(z\)，经过 sigmoid 得到概率 \(\sigma(z)\)（多类时是 softmax，16.1 节）。最大似然给出交叉熵 \(\hl{ce}{-\log\sigma(z)}\)（真实标签 \(y=1\) 时）。你也可以图省事，对概率用平方误差 \(\hl{sqp}{(\sigma(z)-y)^2}\)，两者的最小值都在 \(\sigma(z)=1\)。区别在于模型错得很自信时：比如 \(z=-6\)，模型以 99.75% 的把握说“不会成功”而实际成功了。交叉熵对 \(z\) 的梯度是 \(\sigma(z)-y\approx-1\)，推力最大；平方误差的梯度多了一个因子 \(\sigma(z)(1-\sigma(z))\approx0.0025\)，几乎推不动：sigmoid 在饱和区是平的，平方误差把这种平坦原样传给了梯度，交叉熵的对数恰好把它抵消掉。

A classifier outputs a logit \(z\) and turns it into a probability \(\sigma(z)\) with the sigmoid (softmax for several classes, Lesson 16.1). Maximum likelihood gives the cross-entropy \(\hl{ce}{-\log\sigma(z)}\) (for the true label \(y=1\)). You could also take a shortcut and use the squared error on the probability, \(\hl{sqp}{(\sigma(z)-y)^2}\); both are minimized at \(\sigma(z)=1\). The difference shows when the model is confidently wrong: say \(z=-6\), so the model says "will fail" with 99.75% confidence and the grasp actually succeeded. The gradient of cross-entropy with respect to \(z\) is \(\sigma(z)-y\approx-1\), the strongest push there is; the gradient of the squared error carries an extra factor \(\sigma(z)(1-\sigma(z))\approx0.0025\) and barely pushes at all: the sigmoid is flat when saturated, the squared error passes that flatness on to the gradient, and the logarithm in the cross-entropy cancels it exactly.

第二个任务揭示了交叉熵的另一面：对于可分的数据，它的最小值在 \(z\to\infty\)，模型会被一直推向“100% 确定”，权重越来越大。标签平滑（label smoothing）把目标从 1 改成 \(1-\varepsilon+\varepsilon/K\)（\(K\) 类，图里 \(K=2\)，即 \(1-\varepsilon/2\)），最优 logit 就成了有限值 \(z^\ast=\log\frac{1-\varepsilon/2}{\varepsilon/2}\)。这是一种正则化：它阻止模型过度自信，对标注噪声也更宽容；代价是预测概率被系统性地压低，需要校准好的概率时要小心。

The second task reveals the other side of cross-entropy: on separable data its minimum is at \(z\to\infty\), so the model keeps being pushed towards "100% sure" and its weights keep growing. Label smoothing changes the target from 1 to \(1-\varepsilon+\varepsilon/K\) (\(K\) classes; \(K=2\) in the figure, i.e. \(1-\varepsilon/2\)), and the best logit becomes the finite \(z^\ast=\log\frac{1-\varepsilon/2}{\varepsilon/2}\). This is a regularizer: it stops the model from becoming overconfident and is more forgiving of label noise; the price is that predicted probabilities are pulled down systematically, so be careful where calibrated probabilities matter.

一个直观的比喻：学生拿到去年的考卷和答案。背答案，去年的卷子能考满分，换一套题就不行；理解了方法，两套卷子都考得好。训练集就是去年的卷子，验证集是你自己留出来、训练时从不用于更新参数的另一套题，测试集是最后才打开一次的真正考试。16.1 节已经引入了训练/测试的划分；这里多出的验证集用来做所有“选择”：训练多久、正则化多强、用哪个模型。测试集只能用来报告最终结果，一旦你根据它做了选择，它就不再是公正的估计。

下图在你的浏览器里现场训练一个两层隐藏层、每层 32 个 tanh 单元的网络（1153 个参数），去拟合一个关节的摩擦力矩随速度变化的曲线（库仑摩擦 + 低速的 Stribeck 峰 + 粘性摩擦），只有 12 个带噪声（σ = 0.15 N·m）的测量点。用 Adam（学习率 0.01），每一步用全部训练点（多于 16 个时随机取 16 个）。

A familiar analogy: a student gets last year's exam with the answers. Memorize the answers and last year's paper scores 100%, but a new paper does not; understand the method and both papers go well. The training set is last year's paper; the validation set is another paper you set aside yourself and never use to update parameters; the test set is the real exam, opened once at the very end. Lesson 16.1 introduced the train/test split; the extra validation set is where every "choice" is made: how long to train, how strong the regularization, which model. The test set is only for reporting the final result: once you have made a choice based on it, it is no longer an unbiased estimate.

The figure below trains, live in your browser, a network with two hidden layers of 32 tanh units (1153 parameters) to fit the friction torque of a joint as a function of its velocity (Coulomb friction + a Stribeck peak at low speed + viscous friction), from only 12 noisy (σ = 0.15 N·m) measurements. It uses Adam (learning rate 0.01), each step on all training points (16 random ones when there are more than 16).

注意默认运行里发生的事：到第 2000 步，训练损失只有约 \(2\times10^{-5}\)，网络几乎精确地穿过了每一个训练点，包括它们的噪声；琥珀色曲线在点与点之间剧烈弯曲。验证损失约 0.066，是最低时的 2.6 倍；用真实函数衡量，当前网络的误差（约 0.03）是早停点（约 0.004）的 7 倍左右。这就是过拟合：训练误差和真实误差之间的差距（泛化差距）在扩大。

几种补救方法的效果（同样 12 个点，训练 5000 步，看第 4000 到 5000 步验证损失的平均值除以其最低值）：不加正则化 2.4；L2 系数 \(10^{-4}\) 只是推迟了过拟合（第 3000 步时 1.3，第 5000 步时 2.4），\(3\times10^{-4}\)、\(10^{-3}\) 分别为 1.17、1.08；dropout p = 0.1 或 0.2 约 1.3；输入抖动 0.05 或 0.1（每一步给训练输入加一点高斯噪声，这是 1 维输入上最简单的数据增广）约 1.3；训练点加到 48 个约 1.3，96 个约 1.16。带随机性的方法（dropout、抖动、多于 16 个点时的小批量）会让验证损失一直小幅抖动，所以读数里同时给出“现在”和“最近 1000 步平均”。早停是最便宜的一种：按“早停：恢复最佳权重”，就得到绿色虚线那个网络。

Notice what happens in the default run: by step 2000 the training loss is only about \(2\times10^{-5}\); the network passes almost exactly through every training point, noise included, and the amber curve bends wildly between them. The validation loss is about 0.066, 2.6 times its minimum; measured against the true function, the current network's error (about 0.03) is about 7 times that at the early-stopping point (about 0.004). This is overfitting: the gap between the training error and the true error (the generalization gap) is widening.

How the remedies fare (same 12 points, 5000 steps, mean validation loss over steps 4000 to 5000 divided by its minimum): no regularization 2.4; an L2 weight of \(10^{-4}\) only delays overfitting (1.3 at step 3000, 2.4 at step 5000), \(3\times10^{-4}\) and \(10^{-3}\) give 1.17 and 1.08; dropout p = 0.1 or 0.2: about 1.3; input jitter 0.05 or 0.1 (a little Gaussian noise added to the training inputs at every step, the simplest data augmentation for a 1D input): about 1.3; 48 training points: about 1.3, 96: about 1.16. Methods with randomness (dropout, jitter, minibatches once there are more than 16 points) keep the validation loss jittering a little, which is why the readout shows both "now" and "last-1000-step mean". Early stopping is the cheapest of all: press "Early stop: restore best" and you get the network of the dashed green curve.

为什么模型越灵活越容易过拟合？设想你可以从同一个分布里抽出很多个不同的训练集，每个都拟合一次。偏差是这些拟合的平均离真实函数有多远：模型太僵硬（比如用直线拟合 S 形曲线），平均下来也是错的。方差是这些拟合彼此差多少：模型太灵活，就会跟着每个训练集特有的噪声走，换一组数据就换一个样子。平方误差恰好能分解成三部分：

Why does a more flexible model overfit more easily? Imagine you could draw many different training sets from the same distribution and fit each once. The bias is how far the average of these fits is from the true function: a model that is too rigid (a straight line through an S-shaped curve) is wrong even on average. The variance is how much the fits differ from each other: a model that is too flexible follows the noise peculiar to each training set and changes shape with every new set. The squared error splits exactly into three parts:

左边是期望测试误差，期望是对“抽一个训练集”（以及新样本的噪声）取的；右边第一项是测量噪声（不可约），第二项是偏差的平方，第三项是方差（推导见第 7 节）。下图用多项式回归把它们算出来：多项式的次数就是“容量”旋钮。它是线性最小二乘，有闭式解，所以可以一次对 40 个训练集各拟合一遍。

The left side is the expected test error, the expectation being over "draw a training set" (and the noise of the new example); on the right, the first term is the measurement noise (irreducible), the second the squared bias, the third the variance (derivation in Section 7). The figure below computes them for polynomial regression, with the polynomial degree as the "capacity" knob. It is linear least squares with a closed-form solution, so we can fit each of 40 training sets at once.

λ = 0 时曲线是教科书式的 U 形：次数 0 和 1 偏差很大（直线拟合不了摩擦在零速附近的跳变），次数 5 的期望测试误差最低（0.053），次数 9 时方差已有 0.40，是偏差²（0.025）的十几倍，次数 12 时测试误差超过 20。第二个任务的结论更重要：一个灵活的模型加上合适的正则化，胜过最好的小模型；次数 9 到 12 加上 λ = 0.01，测试误差约 0.042 到 0.045。这正是深度学习的常态：用比需要更大的网络，再用正则化和足够的数据把方差压下去。

交叉验证是现实里估计测试误差的办法，因为你只有一个数据集，不能抽 40 个。\(k\) 折交叉验证把数据分成 \(k\) 份，轮流用 \(k-1\) 份训练、剩下 1 份评估，再取平均（第 7 节）。紫色虚线就是它：它比真正的测试误差噪声大得多，但大致跟着 U 形走；在数据集 #1 上它选了次数 3，测试误差 0.058，离最好的 0.053 不远。按“换一个数据集做 CV”看它怎样随数据抖动。

At λ = 0 the curve is the textbook U: degrees 0 and 1 have a large bias (a line cannot follow the jump of friction near zero speed), degree 5 has the lowest expected test error (0.053), at degree 9 the variance is already 0.40, more than ten times the bias² (0.025), and at degree 12 the test error exceeds 20. The conclusion of the second task matters more: a flexible model with the right regularization beats the best small model; degrees 9 to 12 with λ = 0.01 reach a test error of about 0.042 to 0.045. This is the normal situation in deep learning: use a bigger network than needed, then keep the variance down with regularization and enough data.

Cross-validation is how you estimate the test error in reality, where you have one dataset, not 40. \(k\)-fold cross-validation splits the data into \(k\) parts, trains on \(k-1\) of them and evaluates on the remaining one, in turn, and averages (Section 7). The dashed purple curve is that: much noisier than the true test error, but roughly following the U; on dataset #1 it picks degree 3, with test error 0.058, not far from the best 0.053. Press "Another dataset for CV" to see how it jitters with the data.

正则化就是在训练损失后面加一项惩罚 \(\lambda\,\Omega(\boldsymbol\theta)\)。4.6 节说过它就是一个先验：\(\hl{pen}{\Omega=\|\boldsymbol\theta\|_2^2}\)（L2，岭回归，权重衰减）对应高斯先验，\(\Omega=\|\boldsymbol\theta\|_1\)（L1，lasso）对应拉普拉斯先验。两者都让参数变小，但方式完全不同。几何上看，带惩罚的最小化等价于“在一个范数球里找损失最低的点”（λ 越大，球越小）：L2 的球是圆的，损失的等高椭圆一般在某个没有坐标为零的地方碰到它；L1 的球是一个菱形，有尖角在坐标轴上，椭圆往往先碰到尖角，于是某些坐标恰好等于零。

Regularization adds a penalty \(\lambda\,\Omega(\boldsymbol\theta)\) to the training loss. Lesson 4.6 showed that this is a prior: \(\hl{pen}{\Omega=\|\boldsymbol\theta\|_2^2}\) (L2, ridge, weight decay) corresponds to a Gaussian prior and \(\Omega=\|\boldsymbol\theta\|_1\) (L1, lasso) to a Laplace prior. Both make parameters smaller, but in completely different ways. Geometrically, penalized minimization is equivalent to "find the lowest-loss point inside a norm ball" (larger λ, smaller ball): the L2 ball is round, and the elliptical loss contours usually touch it where no coordinate is zero; the L1 ball is a diamond with corners on the axes, the ellipse tends to hit a corner first, and some coordinates come out exactly zero.

深度学习里显式的 L1/L2 只是一部分，更常见的是下面这些，FIG 3 里试过其中三种：

• 权重衰减：每步把权重乘以略小于 1 的数。对普通 SGD 这与 L2 惩罚等价，对 Adam 不等价（第 7 节），所以现在通常用 AdamW 的“解耦”权重衰减。
• 早停：用验证集挑训练得最好的那一刻。对二次损失上的梯度下降，训练 \(t\) 步大致等价于 \(\lambda\approx1/(\eta t)\) 的 L2（第 7 节）：训练时间本身就是一个正则化旋钮。
• Dropout：训练时每个隐藏单元以概率 \(p\) 被随机置零，留下的乘以 \(1/(1-p)\)；测试时全部保留。网络不能依赖任何单个单元，相当于训练了大量共享权重的子网络的集成。
• 数据增广：对输入做不改变标签的变换，制造“免费”的新样本。对机器人图像是随机裁剪、平移、颜色抖动；对 1 维输入就是 FIG 3 的输入抖动，训练时加小噪声在一定条件下等价于一种平滑正则（Tikhonov 正则化，Bishop 1995）。增广编码了你对不变性的先验知识，常常是最有效的一种。
• 标签平滑：FIG 2，限制分类器的自信程度。
• SGD 的噪声：6.4 节的小批量噪声本身也有正则化作用，这也是极大批量训练常常泛化更差、需要重新调学习率的原因之一。

In deep learning explicit L1/L2 are only part of the story; more common are the following, three of which you tried in FIG 3:

• Weight decay: multiply the weights by a number slightly below 1 at every step. For plain SGD this is equivalent to an L2 penalty, for Adam it is not (Section 7), which is why AdamW's "decoupled" weight decay is the usual choice today.
• Early stopping: use the validation set to pick the best moment of training. For gradient descent on a quadratic loss, training for \(t\) steps is roughly equivalent to L2 with \(\lambda\approx1/(\eta t)\) (Section 7): training time itself is a regularization knob.
• Dropout: during training each hidden unit is zeroed at random with probability \(p\) and the survivors are multiplied by \(1/(1-p)\); at test time all units are kept. The network cannot rely on any single unit, which amounts to training a huge ensemble of weight-sharing subnetworks.
• Data augmentation: apply label-preserving transformations to the inputs to make "free" new examples. For robot images: random crops, shifts, colour jitter; for a 1D input: the input jitter of FIG 3, and training with small input noise is, under some conditions, equivalent to a smoothness regularizer (Tikhonov regularization, Bishop 1995). Augmentation encodes your prior knowledge of invariances and is often the most effective of all.
• Label smoothing: FIG 2; limits how confident a classifier can get.
• SGD noise: the minibatch noise of Lesson 6.4 regularizes too, one reason why very large batches often generalize worse and need the learning rate retuned.

按照 U 形曲线，参数远多于数据的现代网络应该过拟合得一塌糊涂，可它们常常泛化得很好。一个可以完全看清楚的例子：用 \(p\) 个随机 ReLU 特征 \(\phi_j(x)=\max(0,s_j(x-t_j))\)（随机的拐点 \(t_j\) 和方向 \(s_j\)）做线性回归，只学最后一层权重。\(p<n\) 时用最小二乘；\(p>n\) 时有无穷多组权重能精确穿过所有训练点，我们取其中范数最小的那一组 \(\hl{minnorm}{\mathbf w=\Phi^{+}\mathbf y}\)（1.13 节的伪逆）。测试误差随 \(p\) 先降、在 \(\hl{thresh}{p\approx n}\) 处冲上一个尖峰、然后再次下降，这叫双下降。

By the U-shaped curve, modern networks with far more parameters than data should overfit terribly, yet they often generalize well. An example you can see completely: linear regression on \(p\) random ReLU features \(\phi_j(x)=\max(0,s_j(x-t_j))\) (random kinks \(t_j\) and directions \(s_j\)), learning only the last-layer weights. For \(p<n\) use least squares; for \(p>n\) infinitely many weight vectors pass exactly through all training points, and we take the one with the smallest norm, \(\hl{minnorm}{\mathbf w=\Phi^{+}\mathbf y}\) (the pseudo-inverse of Lesson 1.13). The test error first falls as \(p\) grows, shoots up to a spike at \(\hl{thresh}{p\approx n}\), and then falls again: double descent.

数字：经典区（\(p<n\)）里平均误差最低约 0.054（\(p=6\)）；\(p=15\) 时平均误差约 8600，\(p\ge100\) 时回落到 0.017 到 0.019。看紫色小点就会发现，平均值的巨大尖峰几乎全来自 20 次抽样中的一次（它在 \(p=15\) 时误差约 \(10^5\)）；中位数（典型的一次抽样）也有一个温和得多的鼓包：从 \(p=8\) 时的 0.027 升到 \(p=12\) 到 13 时的 0.08，再降到 \(p\ge100\) 时的 0.013，不到它在经典区最好值的一半。为什么会有尖峰？\(p\approx n\) 时特征矩阵 \(\Phi\) 接近方阵，它的最小奇异值可能很小，为了精确穿过每一个带噪声的点，权重必须非常大（方差随 \(\sum 1/s_i^2\) 增长，第 7 节）；\(p\) 再大，能穿过所有点的解有很多，最小范数的那个最“平缓”，方差反而下降。一点点岭正则化（λ = 10⁻⁵）就把尖峰抹平，这是实践中很少看到尖峰的一个原因：几乎总有某种显式或隐式的正则化。深度网络里也观察到了随模型大小、甚至随训练轮数出现的双下降（Nakkiran 等，2020）。本节只点到为止；它说明“参数个数”并不是衡量容量的好指标，有效容量还取决于优化器隐式偏好的那类解。

The numbers: the lowest mean error in the classical regime (\(p<n\)) is about 0.054 (\(p=6\)); at \(p=15\) the mean error is about 8600, and for \(p\ge100\) it falls back to 0.017 to 0.019. The purple dots show that the huge spike of the mean comes almost entirely from one of the 20 draws (whose error at \(p=15\) is about \(10^5\)); the median (a typical draw) has a much gentler bump: from 0.027 at \(p=8\) up to 0.08 at \(p=12\) to 13, and down to 0.013 for \(p\ge100\), less than half its best in the classical regime. Why the spike? At \(p\approx n\) the feature matrix \(\Phi\) is nearly square and its smallest singular value can be tiny, so passing exactly through every noisy point needs very large weights (the variance grows like \(\sum 1/s_i^2\), Section 7); with larger \(p\) many solutions pass through all points, the minimum-norm one is the "gentlest", and its variance goes down. A tiny ridge (λ = 10⁻⁵) flattens the spike, which is one reason it is rarely seen in practice: there is almost always some explicit or implicit regularization. Double descent has also been observed in deep networks, as a function of model size and even of training epochs (Nakkiran et al., 2020). This section only touches on it; its lesson is that "number of parameters" is a poor measure of capacity, because the effective capacity also depends on which solutions the optimizer implicitly prefers.

模型 \(f_{\boldsymbol\theta}\)，逐样本损失 \(\ell(\hat y,y)\)，数据来自分布 \(p(x,y)\)。我们想要小的风险（期望损失），手里却只有 \(n\) 个样本上的经验风险：

A model \(f_{\boldsymbol\theta}\), a per-example loss \(\ell(\hat y,y)\), data from a distribution \(p(x,y)\). We want a small risk (expected loss), but all we have is the empirical risk over \(n\) examples:

对固定的 \(\boldsymbol\theta\)，\(\hat R\) 是 \(R\) 的无偏估计；但 \(\hat{\boldsymbol\theta}\) 是用同一批数据挑出来的，挑选过程偏向那些恰好在这批数据上表现好的参数，所以 \(\hat R(\hat{\boldsymbol\theta})\) 系统性地偏低，泛化差距 \(\Delta\) 为正。验证集上的损失 \(\hl{val}{\hat R_{\mathrm{val}}}\) 用的是没参与挑选的数据，因此是 \(R\) 的公正估计；可一旦你用它来挑选（早停、调 λ），它也会开始偏低，只是偏得少得多，这就是还要留一个测试集的原因。

For a fixed \(\boldsymbol\theta\), \(\hat R\) is an unbiased estimate of \(R\); but \(\hat{\boldsymbol\theta}\) was chosen using the same data, and the choice favours parameters that happen to do well on this particular sample, so \(\hat R(\hat{\boldsymbol\theta})\) is biased low and the generalization gap \(\Delta\) is positive. The validation loss \(\hl{val}{\hat R_{\mathrm{val}}}\) uses data that took no part in the choice, so it is an unbiased estimate of \(R\); once you use it to choose (early stopping, tuning λ) it starts to be biased low too, only much less, which is why a test set is kept as well.

右边的最优条件说明了“拉力”的含义：每个点按 \(\psi(r_i)\) 对参数施力，所有力平衡时就是解。写成 \(\psi(r_i)=w_i r_i\)，\(w_i=\psi(r_i)/r_i\)，就是带权最小二乘的条件，交替更新权重和解就是 FIG 1 用的 IRLS（6.9 节）。平方损失前面的 \(\tfrac12\) 和 MSE 的 \(\tfrac1n\) 只改变尺度，不改变最小值点。

The optimality condition on the right is what "pull" means: each point acts on the parameters with force \(\psi(r_i)\), and the solution is where all forces balance. Writing \(\psi(r_i)=w_i r_i\) with \(w_i=\psi(r_i)/r_i\) turns it into the condition of weighted least squares, and alternating between the weights and the solution is the IRLS used in FIG 1 (Lesson 6.9). The \(\tfrac12\) in front of the squared loss and the \(\tfrac1n\) of the MSE change the scale only, not the minimizer.

分类。\(K\) 类，logit \(\mathbf z\)，概率 \(\mathbf p=\operatorname{softmax}(\mathbf z)\)，目标分布 \(\mathbf y\)（one-hot 或平滑后的）。交叉熵与它的梯度（4.11 节的交叉熵，16.1 节的 softmax）：

Classification. \(K\) classes, logits \(\mathbf z\), probabilities \(\mathbf p=\operatorname{softmax}(\mathbf z)\), a target distribution \(\mathbf y\) (one-hot or smoothed). The cross-entropy and its gradient (cross-entropy from Lesson 4.11, softmax from Lesson 16.1):

交叉熵的梯度就是“预测减目标”，在完全错误时大小接近 1；平方误差多出的 \(\sigma(1-\sigma)\) 在饱和区趋于 0，这就是 FIG 2 里 85 步对 6 步的来源。标签平滑：\(\mathbf y_\varepsilon=(1-\varepsilon)\mathbf e_c+\varepsilon\mathbf 1/K\)。梯度 \(\mathbf p-\mathbf y_\varepsilon=0\) 在 \(\mathbf p=\mathbf y_\varepsilon\) 处成立，即正确类与其他类的 logit 之差为

The cross-entropy gradient is just "prediction minus target", with magnitude close to 1 when the prediction is completely wrong; the extra \(\sigma(1-\sigma)\) of the squared error goes to 0 when saturated, which is where FIG 2's 85 steps versus 6 come from. Label smoothing: \(\mathbf y_\varepsilon=(1-\varepsilon)\mathbf e_c+\varepsilon\mathbf 1/K\). The gradient \(\mathbf p-\mathbf y_\varepsilon\) vanishes at \(\mathbf p=\mathbf y_\varepsilon\), i.e. when the logit of the correct class exceeds the others by

固定一个输入 \(x\)。\(y=f(x)+\epsilon\)，噪声 \(\mathbb E\epsilon=0\)、\(\operatorname{Var}\epsilon=\sigma^2\)，且与训练集无关；\(\hat f\) 依赖于随机的训练集，记 \(\bar f=\mathbb E[\hat f(x)]\)。把 \(y-\hat f\) 拆成三段 \(\epsilon+(f-\bar f)+(\bar f-\hat f)\) 再平方取期望，三个交叉项都为零（\(\epsilon\) 均值为零且与 \(\hat f\) 独立；\(f-\bar f\) 是常数而 \(\mathbb E[\bar f-\hat f]=0\)），于是剩下

Fix an input \(x\). \(y=f(x)+\epsilon\) with noise \(\mathbb E\epsilon=0\), \(\operatorname{Var}\epsilon=\sigma^2\), independent of the training set; \(\hat f\) depends on the random training set; write \(\bar f=\mathbb E[\hat f(x)]\). Split \(y-\hat f\) into the three pieces \(\epsilon+(f-\bar f)+(\bar f-\hat f)\), square and take expectations; all three cross terms vanish (\(\epsilon\) has mean zero and is independent of \(\hat f\); \(f-\bar f\) is a constant and \(\mathbb E[\bar f-\hat f]=0\)), leaving

FIG 4 把它在 81 个均匀分布的 \(x\) 上平均，期望用 40 个训练集的平均近似。分解只对平方损失这么干净；对 0-1 损失或交叉熵有类似但不可加的版本。\(k\) 折交叉验证把索引分成 \(k\) 组 \(I_1,\dots,I_k\)，\(\hat f^{(-j)}\) 是不用第 \(j\) 组训练出的模型：

FIG 4 averages this over 81 evenly spaced \(x\), approximating the expectation by the average over 40 training sets. The decomposition is this clean only for the squared loss; for the 0-1 loss or cross-entropy there are similar but non-additive versions. \(k\)-fold cross-validation splits the indices into \(k\) groups \(I_1,\dots,I_k\); \(\hat f^{(-j)}\) is the model trained without group \(j\):

对线性最小二乘 \(\frac1{2n}\|X\mathbf w-\mathbf y\|^2\)，记 \(A=X^\top X/n\)、\(\hat{\mathbf w}\) 为无正则化的解，则损失等于 \(\tfrac12(\mathbf w-\hat{\mathbf w})^\top A(\mathbf w-\hat{\mathbf w})\) 加常数（FIG 5 用的就是这个形式）。加上 \(\tfrac\lambda2\|\mathbf w\|_2^2\)，令梯度为零：

For linear least squares \(\frac1{2n}\|X\mathbf w-\mathbf y\|^2\), with \(A=X^\top X/n\) and \(\hat{\mathbf w}\) the unregularized solution, the loss equals \(\tfrac12(\mathbf w-\hat{\mathbf w})^\top A(\mathbf w-\hat{\mathbf w})\) plus a constant (the form FIG 5 uses). Add \(\tfrac\lambda2\|\mathbf w\|_2^2\) and set the gradient to zero:

L2 把每个特征方向按 \(d_i/(d_i+\lambda)\) 收缩：数据约束强的方向（大 \(d_i\)）几乎不动，数据几乎不约束的方向（小 \(d_i\)，方差的主要来源）被压向零，但永远不会恰好为零。L1 不可导，用次梯度：在 \(A\) 为对角阵（特征正交）时问题按坐标分开，每个坐标的最优条件 \(d_i(w_i-\hat w_i)+\lambda\,\partial|w_i|\ni0\) 给出软阈值

L2 shrinks each feature direction by \(d_i/(d_i+\lambda)\): directions the data constrain strongly (large \(d_i\)) barely move, directions the data hardly constrain (small \(d_i\), the main source of variance) are pushed towards zero, but never exactly to zero. L1 is not differentiable, so use subgradients: when \(A\) is diagonal (orthogonal features) the problem separates by coordinate, and the optimality condition \(d_i(w_i-\hat w_i)+\lambda\,\partial|w_i|\ni0\) of each coordinate gives the soft threshold

只要 \(|\hat w_i|\le\lambda/d_i\)，解就精确为零：这就是稀疏性的代数来源，对应几何上的菱形尖角。特征相关时没有闭式解，FIG 5 用坐标下降：反复对每个坐标做上面这个软阈值，其他坐标暂时固定。lasso 的已知弱点也在这里：一组高度相关的有用特征，它往往只留一个、随意丢掉其余的。

Whenever \(|\hat w_i|\le\lambda/d_i\), the solution is exactly zero: this is the algebraic source of sparsity, matching the corners of the diamond. With correlated features there is no closed form, and FIG 5 uses coordinate descent: apply this soft threshold to each coordinate in turn with the others held fixed, and repeat. The lasso's known weakness shows here too: from a group of highly correlated useful features it tends to keep one and drop the rest more or less arbitrarily.

对上面的二次损失从 \(\mathbf w=0\) 出发做梯度下降（学习率 \(\eta\)），沿第 \(i\) 个特征方向第 \(t\) 步的分量是 \(\big(1-(1-\eta d_i)^t\big)\,\hat w_i\)。与 L2 的 \(d_i/(d_i+\lambda)\) 比较：两者都让大 \(d_i\) 方向先学到、小 \(d_i\) 方向被抑制；当 \(\eta d_i\ll1\) 时，\(t\) 步的早停大致相当于 \(\lambda\approx1/(\eta t)\)（Goodfellow 等《深度学习》7.8 节）。所以 FIG 3 里验证损失先降后升，是因为网络先学到了数据支持强的平滑部分，后学到的是被噪声主导的方向。

权重衰减 ≠ L2（对 Adam）。L2 惩罚把 \(2\lambda\boldsymbol\theta\) 加进梯度；SGD 的一步 \(\boldsymbol\theta\leftarrow\boldsymbol\theta-\eta(\mathbf g+2\lambda\boldsymbol\theta)=(1-2\eta\lambda)\boldsymbol\theta-\eta\mathbf g\) 正好就是“先衰减再走一步”。Adam 却会把整个梯度除以它的二阶矩估计（6.3、6.4 节），于是 \(2\lambda\boldsymbol\theta\) 这一项也被逐坐标缩放，梯度大的参数几乎不被衰减。AdamW（Loshchilov & Hutter, 2019）把衰减从梯度里拿出来单独做：\(\boldsymbol\theta\leftarrow\boldsymbol\theta-\eta\,(\mathbf u_t+\lambda\boldsymbol\theta)\)，其中 \(\mathbf u_t\) 是 Adam 本来要走的那一步（只由数据损失的梯度算出）。FIG 3 用的是写进损失的 L2 惩罚（加在梯度里，再交给 Adam），这在我们这里也足够有效。

Dropout：训练时隐藏单元输出 \(h_j\) 换成 \(m_j h_j/(1-p)\)，\(m_j\sim\mathrm{Bernoulli}(1-p)\)，于是 \(\mathbb E[m_jh_j/(1-p)]=h_j\)，测试时直接用 \(h_j\)，期望不变（“inverted dropout”）。

Run gradient descent with learning rate \(\eta\) on the quadratic above, starting from \(\mathbf w=0\); after \(t\) steps the component along feature direction \(i\) is \(\big(1-(1-\eta d_i)^t\big)\,\hat w_i\). Compare with L2's \(d_i/(d_i+\lambda)\): both let large-\(d_i\) directions be learned first and hold back small-\(d_i\) ones; for \(\eta d_i\ll1\), stopping after \(t\) steps is roughly \(\lambda\approx1/(\eta t)\) (Goodfellow et al., Deep Learning, Section 7.8). That is why FIG 3's validation loss falls and then rises: the network learns the smooth part strongly supported by the data first and the noise-dominated directions later.

Weight decay ≠ L2 (for Adam). An L2 penalty adds \(2\lambda\boldsymbol\theta\) to the gradient; an SGD step \(\boldsymbol\theta\leftarrow\boldsymbol\theta-\eta(\mathbf g+2\lambda\boldsymbol\theta)=(1-2\eta\lambda)\boldsymbol\theta-\eta\mathbf g\) is exactly "decay, then step". Adam, however, divides the whole gradient by its second-moment estimate (Lessons 6.3 and 6.4), so the \(2\lambda\boldsymbol\theta\) term is rescaled per coordinate as well, and parameters with large gradients are barely decayed. AdamW (Loshchilov & Hutter, 2019) takes the decay out of the gradient and applies it separately: \(\boldsymbol\theta\leftarrow\boldsymbol\theta-\eta\,(\mathbf u_t+\lambda\boldsymbol\theta)\), where \(\mathbf u_t\) is the step Adam would take (computed from the gradient of the data loss only). FIG 3 uses an L2 penalty written into the loss (added to the gradient, then handed to Adam), which is effective enough here.

Dropout: during training a hidden output \(h_j\) is replaced by \(m_j h_j/(1-p)\) with \(m_j\sim\mathrm{Bernoulli}(1-p)\), so \(\mathbb E[m_jh_j/(1-p)]=h_j\) and at test time \(h_j\) is used as is, with the same expectation ("inverted dropout").

\(\Phi\in\R^{n\times p}\) 的奇异值分解 \(\Phi=\sum_i s_i\mathbf u_i\mathbf v_i^\top\)。\(p\ge n\) 且满秩时，所有插值解中范数最小的是 \(\mathbf w=\Phi^+\mathbf y=\sum_i \frac{\mathbf u_i^\top\mathbf y}{s_i}\mathbf v_i\)（1.13 节）。观测噪声 \(\boldsymbol\epsilon\) 通过 \(1/s_i\) 被放大，噪声在权重上造成的方差是 \(\sigma^2\sum_i 1/s_i^2\)；\(p\) 刚好等于 \(n\) 附近时最小的 \(s_i\) 最接近零，所以出现尖峰。加岭正则把 \(1/s_i\) 换成 \(s_i/(s_i^2+n\lambda)\)，上限是 \(1/(2\sqrt{n\lambda})\)，尖峰随之消失。

Take the SVD \(\Phi=\sum_i s_i\mathbf u_i\mathbf v_i^\top\) of \(\Phi\in\R^{n\times p}\). For \(p\ge n\) and full rank, the minimum-norm interpolating solution is \(\mathbf w=\Phi^+\mathbf y=\sum_i \frac{\mathbf u_i^\top\mathbf y}{s_i}\mathbf v_i\) (Lesson 1.13). Observation noise \(\boldsymbol\epsilon\) is amplified by \(1/s_i\), and the variance it causes in the weights is \(\sigma^2\sum_i 1/s_i^2\); near \(p=n\) the smallest \(s_i\) is closest to zero, hence the spike. A ridge replaces \(1/s_i\) by \(s_i/(s_i^2+n\lambda)\), which is at most \(1/(2\sqrt{n\lambda})\), and the spike disappears.

上面所有的保证都假设训练数据和测试数据来自同一个分布。协变量偏移是最常见的破坏方式（下标 tr 表示训练，te 表示测试）：\(p(y\mid x)\) 不变（物理规律没变），但输入的分布从 \(p_{\mathrm{tr}}(x)\) 变成了 \(\hl{shift}{p_{\mathrm{te}}(x)}\)（机器人去了没去过的状态）。测试风险可以改写成训练分布上的加权期望：

Every guarantee above assumes that training and test data come from the same distribution. Covariate shift is the most common way this breaks (subscripts tr for training, te for test): \(p(y\mid x)\) stays the same (the physics has not changed), but the input distribution moves from \(p_{\mathrm{tr}}(x)\) to \(\hl{shift}{p_{\mathrm{te}}(x)}\) (the robot goes to states it has not visited). The test risk can be rewritten as a weighted expectation over the training distribution:

这就是重要性加权（Shimodaira, 2000），前提是 \(p_{\mathrm{tr}}(x)>0\) 处处覆盖 \(p_{\mathrm{te}}\)。如果测试分布落在训练分布完全没有覆盖的地方，权重是无穷大，任何重新加权都救不了：模型在那里的预测只取决于它的结构偏好（外推），数据不提供任何约束。下一节就是这种情形。

This is importance weighting (Shimodaira, 2000), and it requires \(p_{\mathrm{tr}}(x)>0\) wherever \(p_{\mathrm{te}}\) has mass. If the test distribution lands where the training distribution has no coverage at all, the weight is infinite and no reweighting can help: the model's predictions there depend only on its structural preferences (extrapolation), with no constraint from data. The next section is exactly that case.

一台机械臂要学一个关节的重力补偿力矩 \(\tau_g(q)=2\sin q\)（N·m）。采数据时为了安全，只在 \(|q|\le0.8\) rad 的小范围里动，得到 40 个带噪声（0.05 N·m）的点；在这个范围里 \(\sin q\) 几乎是一条直线。训练 5 个初值不同的网络（深度集成，各训练 800 步），然后让机器人去 \(q\approx2\) rad 的位置工作。拖动下方红色的测试分布，看误差和集成的分歧怎样变化。

A robot arm is to learn the gravity-compensation torque of one joint, \(\tau_g(q)=2\sin q\) (N·m). For safety, data were collected only in the small range \(|q|\le0.8\) rad, giving 40 noisy (0.05 N·m) points; over this range \(\sin q\) is almost a straight line. Train 5 networks from different initializations (a deep ensemble, 800 steps each), then send the robot to work near \(q\approx2\) rad. Drag the red test distribution below and watch how the error and the disagreement of the ensemble change.

三点观察。(1) 在训练范围里，5 个网络的误差都很小（μ = 0 时均方误差约 \(10^{-4}\)），在范围之外它们各自“合理地”外推，可数据完全不约束这种外推，μ = 2 时均方误差约 0.5，μ = 3 时约 6。(2) 集成的分歧确实随偏移增大（标准差从不到 0.001 增到 μ = 2 时约 0.05），可以当作“我到了没见过的地方”的报警信号（21.3 节），但它严重低估了真实误差：5 个网络犯了相似的错误。(3) 唯一真正有效的办法是到那里去采数据。机器人领域有一整套方法都是这个思路：DAgger 让策略自己跑、在它到达的状态上请专家标注（22.1 节）；域随机化在仿真里把训练分布撑得足够宽，让真实世界落在里面（23.2 节）。

Three observations. (1) Inside the training range all 5 networks are accurate (mean squared error about \(10^{-4}\) at μ = 0); outside it each extrapolates "plausibly", but nothing in the data constrains that extrapolation: the mean squared error is about 0.5 at μ = 2 and about 6 at μ = 3. (2) The ensemble's disagreement does grow with the shift (its std goes from under 0.001 to about 0.05 at μ = 2) and can serve as an "I am somewhere new" alarm (Lesson 21.3), but it badly underestimates the actual error: the 5 networks make similar mistakes. (3) The only remedy that really works is to go there and collect data. A whole family of robotics methods follows that idea: DAgger runs the policy and asks an expert to label the states it reaches (Lesson 22.1); domain randomization widens the training distribution in simulation until the real world falls inside it (Lesson 23.2).