Part I · 第 6 章优化 · 6.4Part I · Ch. 6 Optimization · 6.4

随机梯度下降Stochastic Gradient Descent

真实的代价函数常常是几百万条数据误差的平均。每走一步都把所有数据看一遍，太贵了。随机梯度下降 (SGD) 的想法很朴素：只抽一小撮样本（一个小批量）估计梯度，宁可走得歪一点，也要走得快。这个"歪"，也就是噪声，既带来代价（走到最后停不稳），也带来意外的好处（能逃出鞍点）。 Real cost functions are often averages over millions of data errors, and looking at all of the data at every step is far too expensive. The idea of stochastic gradient descent (SGD) is plain: estimate the gradient from a small random handful of samples (a minibatch) and accept a slightly crooked step in exchange for a much faster one. That crookedness, the noise, has a price (you never settle quite still at the end) and an unexpected benefit (it can shake you out of saddle points).

约 25 分钟~25 min 前置：6.3 梯度下降家族 · 4.3 期望与方差 · 1.13 最小二乘Prereqs: 6.3 Gradient descent family · 4.3 Expectation & variance · 1.13 Least squares 5 个交互5 interactives

1为什么机器人需要它Why robots need this

机器人学习的数据又多又杂：一次遥操作采集就是几十万帧图像加动作，训练一个模仿学习策略的代价函数就是"每一帧上动作预测误差的平均"。如果每次更新参数都要遍历整个数据集，一轮训练可能要几个小时，而参数只更新了一次。

所以训练几乎都用小批量：每次随机抽几十到几千帧，算这些帧上的平均梯度，更新一次，再抽下一批。同样的思路也出现在机器人在线运行时：标定参数一边采数据一边更新，不必等数据采完；每来一个新测量，就沿着它的梯度挪一小步。

这一节回答三个问题：为什么抽一小撮样本算出来的梯度仍然"可用"？噪声对收敛有什么影响，学习率该怎么随时间变化？噪声什么时候反而是朋友？

Robot-learning data is plentiful and messy: one teleoperation session is hundreds of thousands of frames of images and actions, and the cost for training an imitation policy is "the average action-prediction error over every frame". If every parameter update needed a pass over the whole dataset, one epoch could take hours for a single update.

So training nearly always uses minibatches: draw a few dozen to a few thousand frames at random, compute the average gradient over them, update once, then draw the next batch. The same idea shows up when a robot runs online: calibration parameters are updated while data arrives, with no waiting for the full set; each new measurement nudges the estimate along its own gradient.

This lesson answers three questions. Why is a gradient computed from a handful of samples still "usable"? What does the noise do to convergence, and how should the learning rate change over time? And when is noise actually a friend?

2直觉：民意调查Intuition: an opinion poll

想象要估计全国人口的平均身高。把所有人都量一遍最准，但太贵；随机抽 100 个人，算出来的平均值每次都不同，却围绕真值上下波动，抽得越多，波动越小。梯度也是平均值：$\nabla f=\frac1N\sum_i\nabla f_i$，是所有样本梯度的平均。随机抽 $B$ 个样本再平均，就是一次"民意调查"，得到一个有噪声但没有偏向的估计。

"没有偏向"是整件事的支点：噪声会让每一步走歪，但平均而言走的方向是对的。用 SGD 下山，就像一个带着不太准的指南针的徒步者：每一步都偏一点，走一百步，大方向没错。

Imagine estimating the average height of a country. Measuring everyone is most accurate but far too costly; pick 100 people at random and the average differs every time, yet it fluctuates around the truth, less and less as you sample more. The gradient is also an average: $\nabla f=\frac1N\sum_i\nabla f_i$, the mean of the per-sample gradients. Drawing $B$ samples and averaging them is one "opinion poll": an estimate that is noisy but unbiased.

"Unbiased" is the pivot of the whole subject: the noise makes every step crooked, but on average the direction is right. Descending with SGD is like hiking with a slightly unreliable compass: every step points a bit off, and after a hundred steps the overall direction is still correct.

3动手玩：SGD 对 GDPlay: SGD versus GD

任务是用一条直线 $y=wu+b$ 拟合 200 个带噪声的点，参数 $\x=(w,b)$，代价是平均平方误差。左图是数据和当前的直线，右图是参数空间的代价地形。紫色是每一步都用全部 200 个点的 GD，琥珀色是每一步只抽 $B$ 个点的 SGD（左图中带圈的点）。两者起点、学习率相同。

The task is to fit a line $y=wu+b$ to 200 noisy points, parameters $\x=(w,b)$, cost the mean squared error. The left panel is the data and the current lines; the right panel is the cost landscape in parameter space. Purple is GD using all 200 points at every step; amber is SGD using only $B$ points per step (the circled points on the left). Both start at the same place with the same learning rate.

FIG 1小批量梯度下降Minibatch gradient descent

右图：黄色箭头是这一步小批量算出的 $-\eta\,\mathbf g_B$，青色箭头是真正的 $-\eta\nabla f$。下图是两者代价与最优值之差 $f-f^\star$（对数坐标，横轴是更新次数）。小批量按"打乱后依次取"的方式抽取：$B=200$ 时一批就是全部数据，SGD 与 GD 完全重合。可点击右图设置新的起点。 Right: the amber arrow is this step's minibatch $-\eta\,\mathbf g_B$; the cyan arrow is the true $-\eta\nabla f$. Bottom: $f-f^\star$ for both (log scale, horizontal axis = number of updates). Batches are drawn by "shuffle, then take in turn"; with $B=200$ one batch is the whole dataset and SGD coincides with GD. Click the right panel to set a new start.

试一试Try this

设 $B=1$ 运行 100 步以上：路径在最小值附近"抖"成一团，永远停不下来。Set $B=1$ and run 100+ steps: the path jitters in a cloud around the minimum and never settles.
把 $B$ 拉到 200：两条路径完全重合，因为这时 SGD 就是 GD。Push $B$ to 200: the two paths coincide, because SGD is GD then.
保持 $B=1$，把 $\eta$ 调小到 0.01 以下再跑：抖动变小了，但前进变慢。这是噪声与速度的权衡。Keep $B=1$ and lower $\eta$ below 0.01: the jitter shrinks but progress slows. That is the trade-off between noise and speed.

核心洞见Key insight

SGD 每一步的成本只有 GD 的 $B/N$，却走得几乎一样"对"。早期（离最优点很远）时，梯度信号远大于噪声，小批量足够；晚期（离最优点很近）时，真梯度很小，噪声占了上风，这就是为什么要逐渐减小学习率或增大批量。

Each SGD step costs only $B/N$ of a GD step yet points almost as well. Early on (far from the optimum) the gradient signal dwarfs the noise and a small batch is enough; late on (near the optimum) the true gradient is tiny and noise wins, which is why you shrink the learning rate or grow the batch over time.

4噪声到底有多大：梯度的"云"How noisy exactly: the cloud of gradients

固定在参数空间的一个点 $\x$，反复抽取小批量，每次得到一个梯度估计 $\mathbf g_B$。把这些估计画在"梯度空间"里，就是一团围绕真梯度 $\nabla f(\x)$ 的点云。点云的中心就是真梯度（无偏），点云的大小随 $B$ 增大而缩小，标准差按 $1/\sqrt B$ 收缩。拖动左图的手柄换一个位置，改变批量，对照理论预测的椭圆。

Fix a point $\x$ in parameter space and draw minibatches over and over, each time getting a gradient estimate $\mathbf g_B$. Plotted in "gradient space", the estimates form a cloud around the true gradient $\nabla f(\x)$. The center of the cloud is the true gradient (unbiasedness), and the cloud shrinks as $B$ grows, with standard deviation falling as $1/\sqrt B$. Drag the handle on the left to move the point, change the batch size, and compare with the predicted ellipse.

FIG 2梯度的点云The gradient cloud

右图：150 次独立抽取的小批量梯度（点），样本均值（空心点）、真梯度 $\nabla f$（青色箭头，从原点出发；原点是"梯度为零"），以及由数据协方差 $\Sigma$ 算出的理论 2σ 椭圆（虚线）。梯度分量与 $\x$ 的位置有关：越靠近最优点，真梯度越短，噪声云却几乎不变小。 Right: 150 independent minibatch gradients (dots), their sample mean (hollow dot), the true gradient $\nabla f$ (cyan arrow from the origin, the "zero gradient" point), and the theoretical 2σ ellipse from the data covariance $\Sigma$ (dashed). Move closer to the optimum and the true gradient gets shorter while the noise cloud barely shrinks.

把 $B$ 从 4 调到 16：点云的大小缩小为约一半（$\sqrt{16/4}=2$）。Go from $B=4$ to $B=16$: the cloud shrinks to about half its size ($\sqrt{16/4}=2$).
把手柄拖到最小值（右下方的最优点 $\x^\star$）附近，设 $B=4$：真梯度几乎被噪声淹没，这时单个小批量甚至可能指向上坡。Drag the handle next to the minimum (the optimum $\x^\star$, to the lower right) with $B=4$: the true gradient is drowned in noise, and a single minibatch may even point uphill.

5学习率日程：先大后小Learning-rate schedules: big first, then small

噪声让 SGD 在最小值附近不断被"踢"开。固定学习率 $\eta$ 时，每一步被踢的幅度 $\propto\eta$，而回拉的力来自曲率，两者平衡下来，参数不会收敛到最优点，而是停在一个噪声球里，球的"代价高度"约为 $\eta\,\mathrm{tr}(C)/(4B)$（$C$ 是单样本梯度在最优点的协方差，后面会推导）。想要更低，要么减小 $\eta$，要么增大 $B$。

实践中的常规做法是让 $\eta$ 随时间减小：开头用大学习率快速接近，结尾用小学习率压低噪声球。下面比较五种常见的日程；每条曲线是 24 次独立运行的平均，每次运行都是真实的 SGD。

Noise keeps kicking SGD away from the minimum. With a constant learning rate $\eta$ each kick has size $\propto\eta$, while the pull-back comes from curvature; at balance the parameters do not converge to the optimum but sit in a noise ball whose "cost height" is about $\eta\,\mathrm{tr}(C)/(4B)$ ($C$ is the covariance of the per-sample gradient at the optimum, derived below). For lower you either decrease $\eta$ or increase $B$.

The standard practice is to let $\eta$ shrink over time: a large rate to approach quickly, a small rate at the end to squash the noise ball. Below are five common schedules; each curve is the average of 24 independent runs, every one of them a real SGD run.

FIG 3五种学习率日程Five learning-rate schedules

左：$\eta_k$ 随步数 $k$ 的变化。右：平均 $f-f^\star$（对数坐标）。虚线是常数日程的理论噪声球高度 $\eta_0\,\mathrm{tr}(C)/(4B)$。日程：常数；$\eta_0/(1+k/50)$；阶梯（$k=T/3$ 与 $2T/3$ 处分别乘以 0.3）；余弦衰减至 0；5% 线性预热后接余弦。$T=600$，数据与 FIG 1 相同，小批量有放回地抽取。 Left: $\eta_k$ against step $k$. Right: mean $f-f^\star$ (log scale). The dashed line is the theoretical noise-ball height $\eta_0\,\mathrm{tr}(C)/(4B)$ for the constant schedule. Schedules: constant; $\eta_0/(1+k/50)$; step (×0.3 at $k=T/3$ and $2T/3$); cosine decay to 0; 5% linear warm-up followed by cosine. $T=600$, same data as FIG 1, batches drawn with replacement.

让余弦日程的最终误差比常数日程低 3 倍以上（默认设置就够，调一下滑块试试它为何稳健）。Get the cosine schedule's final error at least 3× below the constant schedule's (the defaults may already do it; wiggle the sliders to see how robust it is).
不用任何衰减日程，只靠调 $\eta_0$ 或 $B$，让常数日程的最终误差低于 $10^{-3}$。With no decay schedule at all, tune $\eta_0$ or $B$ so the constant schedule ends below $10^{-3}$.
把 $\eta_0$ 调到最大并取 $B=1$：常数日程发散。衰减日程会在学习率降下来之前就被"炸飞"吗？Push $\eta_0$ to the maximum with $B=1$: the constant schedule diverges. Do the decaying schedules blow up before the rate comes down?

6噪声什么时候是朋友：逃出鞍点When noise is a friend: escaping a saddle

6.3 节的"鞍点"地形 $f=\tfrac12x^2+\tfrac14y^4-\tfrac12y^2$：沿 $x$ 方向是碗，沿 $y$ 方向，原点处是山脊（$f_{yy}=-1$），两侧才是 $(0,\pm1)$ 两个谷底。如果起点恰好在 $y=0$ 这条线上，梯度的 $y$ 分量永远是 0，GD 会老老实实滑进鞍点 $(0,0)$ 并停在那里。

现在给每个样本加一个零均值的扰动 $f_i(\x)=f(\x)+\boldsymbol\xi_i^\top\x$（$\sum_i\boldsymbol\xi_i=\mathbf 0$，所以全批量梯度还是 $\nabla f$）。小批量梯度是 $\nabla f(\x)+\bar{\boldsymbol\xi}_B$：噪声自带 $y$ 方向的分量，把 24 个小球从 $y=0$ 线上踢开，山脊的负曲率再把它们推向两个谷之一。

Take the "saddle" terrain of Lesson 6.3, $f=\tfrac12x^2+\tfrac14y^4-\tfrac12y^2$: along $x$ it is a bowl, along $y$ the origin is a ridge ($f_{yy}=-1$), with the two valley floors at $(0,\pm1)$ on either side. If the start lies exactly on the line $y=0$, the $y$-component of the gradient is always 0, and GD obediently slides into the saddle $(0,0)$ and stays there.

Now give each sample a zero-mean perturbation $f_i(\x)=f(\x)+\boldsymbol\xi_i^\top\x$ (with $\sum_i\boldsymbol\xi_i=\mathbf 0$, so the full-batch gradient is still $\nabla f$). The minibatch gradient is $\nabla f(\x)+\bar{\boldsymbol\xi}_B$: the noise carries a $y$-component, kicks the 24 balls off the line $y=0$, and the ridge's negative curvature pushes each toward one of the two valleys.

FIG 424 个小球逃离鞍点24 balls escaping a saddle

下方：各小球的 $y$ 坐标分布Histogram of the balls' $y$

所有小球从同一点 $(-2.4,\,0)$ 出发，每一步用各自独立抽取的小批量。噪声强度 $\sigma$ 是单样本扰动 $\boldsymbol\xi_i$ 的标准差，数据集大小 $N=64$。"GD"模式用全批量，没有噪声；此时 24 个小球完全重合。 All balls start from the same point $(-2.4,\,0)$ and each uses its own independently drawn minibatch every step. The noise strength $\sigma$ is the standard deviation of the per-sample perturbation $\boldsymbol\xi_i$, and the dataset has $N=64$ samples. In "GD" mode the full batch is used, there is no noise, and all 24 balls coincide.

选 "GD"，运行：所有小球都停在鞍点。（梯度为零 ≠ 最小值，6.1 节。）Choose "GD" and run: every ball ends on the saddle. (Zero gradient ≠ minimum, Lesson 6.1.)
选 "SGD"，运行：大部分小球逃出鞍点，分别落入上下两个谷底。Choose "SGD" and run: most balls escape and fall into the upper or lower valley.
逃出之后，小球仍在谷底附近"抖动"。把 $B$ 调大或 $\sigma$ 调小，抖动的幅度怎么变？After escaping, the balls still jitter around the valley floors. How does the jitter change as you raise $B$ or lower $\sigma$?

7再看数学The math

提示：悬停在带底色的符号上，FIG 1、FIG 2 和 FIG 3 中对应的元素（小批量梯度、真梯度、噪声椭圆、噪声球虚线）会被强调。Tip: hover the highlighted symbols; the matching elements of FIG 1, FIG 2 and FIG 3 (minibatch gradient, true gradient, noise ellipse, noise-ball line) light up.

有限和与小批量梯度Finite sums and the minibatch gradient

机器学习与标定里的代价几乎总是样本平均：

Costs in machine learning and calibration are almost always sample averages:

$$ f(\x)=\frac1N\sum_{i=1}^{N}f_i(\x),\qquad \hl{full}{\cx{\nabla f(\x)}}=\frac1N\sum_{i=1}^{N}\nabla f_i(\x),\qquad \hl{mb}{\cv{\mathbf g_B(\x)}}=\frac1B\sum_{i\in\mathcal B}\nabla f_i(\x) $$

其中 $\mathcal B$ 是随机抽出的 $B$ 个下标。若每个下标被抽到的概率相同，则 $\mathbf g_B$ 是 $\nabla f$ 的无偏估计；协方差为

where $\mathcal B$ is a random set of $B$ indices. If every index is equally likely, $\mathbf g_B$ is an unbiased estimator of $\nabla f$, with covariance

$$ \mathbb E[\mathbf g_B]=\nabla f(\x),\qquad \hl{cov}{\mathrm{Cov}[\mathbf g_B]}=\frac{\Sigma(\x)}{B}\cdot\frac{N-B}{N-1}\ \ \Big(\text{with replacement: }\frac{\Sigma(\x)}{B}\Big),\qquad \Sigma(\x)=\frac1N\sum_i\big(\nabla f_i-\nabla f\big)\big(\nabla f_i-\nabla f\big)^{\!\top} $$

$\Sigma$ 是单样本梯度的协方差。因子 $1/B$ 就是 FIG 2 里点云随 $B$ 缩小的原因；$(N-B)/(N-1)$ 是"不放回"抽样的修正，$B=N$ 时为 0（没有随机性）。

$\Sigma$ is the covariance of the per-sample gradient. The factor $1/B$ is why the cloud of FIG 2 shrinks with $B$; $(N-B)/(N-1)$ is the correction for sampling without replacement and vanishes at $B=N$ (no randomness left).

SGD 的更新与收敛条件The SGD update and convergence conditions

$$ \x_{k+1}=\x_k-\eta_k\,\cv{\mathbf g_{B}(\x_k)} $$

要让 SGD 收敛到最优点（凸问题），Robbins–Monro 条件要求：$\sum_k\eta_k=\infty$（走得动：总路程够长）且 $\sum_k\eta_k^2<\infty$（停得住：噪声的累积方差有限）。$\eta_k=\eta_0/(1+k/\tau)$ 这类 $1/k$ 衰减满足条件，常数 $\eta$ 不满足。

For SGD to converge to the optimum (in the convex case), the Robbins–Monro conditions require $\sum_k\eta_k=\infty$ (you can get there: the total path is long enough) and $\sum_k\eta_k^2<\infty$ (you can stop: the accumulated noise variance is finite). A $1/k$ decay such as $\eta_k=\eta_0/(1+k/\tau)$ satisfies them; a constant $\eta$ does not.

噪声球的高度（二次函数上的推导）The height of the noise ball (derivation on a quadratic)

在最优点附近 $f\approx f^\star+\tfrac12\mathbf e^\top H\mathbf e$，$\mathbf e=\x-\x^\star$。一步更新为 $\mathbf e_{k+1}=(I-\eta H)\mathbf e_k-\eta\boldsymbol\zeta_k$，其中噪声 $\boldsymbol\zeta_k$ 均值为 0、协方差 $C_B=C/B$（$C$ 是 $\Sigma$ 在最优点处的值）。取稳态协方差 $P=\mathbb E[\mathbf e\mathbf e^\top]$，令 $P=(I-\eta H)P(I-\eta H)+\eta^2C_B$，展开并略去 $\eta^2$ 项：

Near the optimum $f\approx f^\star+\tfrac12\mathbf e^\top H\mathbf e$ with $\mathbf e=\x-\x^\star$. One update reads $\mathbf e_{k+1}=(I-\eta H)\mathbf e_k-\eta\boldsymbol\zeta_k$, where the noise $\boldsymbol\zeta_k$ has mean 0 and covariance $C_B=C/B$ ($C$ is $\Sigma$ at the optimum). In the stationary state $P=\mathbb E[\mathbf e\mathbf e^\top]$ satisfies $P=(I-\eta H)P(I-\eta H)+\eta^2C_B$; expanding and dropping the $\eta^2$ terms on the left:

$$ HP+PH\approx\eta\,C_B\ \ \Longrightarrow\ \ \mathbb E\big[f-f^\star\big]=\tfrac12\mathrm{tr}(HP)\approx\frac{\eta}{4}\,\mathrm{tr}(C_B)=\hl{floor}{\frac{\eta\,\mathrm{tr}(C)}{4B}} $$

取迹时用到 $\mathrm{tr}(HP+PH)=2\,\mathrm{tr}(HP)$。结论非常干净：噪声球的高度与 $\eta$ 成正比，与 $B$ 成反比。把 $\eta$ 和 $B$ 同时乘以 2 的话，球的高度不变，这就是深度学习里"批量加倍，学习率也加倍"的线性缩放经验（小批量范围内）的来源。FIG 3 的虚线正是这个公式。

Taking the trace uses $\mathrm{tr}(HP+PH)=2\,\mathrm{tr}(HP)$. The conclusion is clean: the noise ball's height is proportional to $\eta$ and inversely proportional to $B$. Doubling $\eta$ and $B$ together leaves it unchanged, the origin of the "double the batch, double the learning rate" rule of thumb in deep learning (in the small-batch regime). The dashed line in FIG 3 is exactly this formula.

动量与 Adam 放在 SGD 里Momentum and Adam in the SGD setting

6.3 节的动量、RMSProp、Adam，在实际训练里都是用小批量梯度代替 $\nabla f$ 来运行的。动量对连续多步的 $\mathbf g_B$ 取指数平均，等于把噪声平均掉了一部分，但它同时把有效学习率放大到约 $\eta/(1-\beta)$，噪声球也按这个有效学习率缩放；Adam 的二阶矩 $\mathbf s$ 里含有噪声的方差，因此在梯度很小、几乎全是噪声的坐标上它会自动把步子变小。这些在 Part V 的"损失、泛化与正则化"和"反向传播"章节里会反复用到；这里只需记住：噪声来自小批量，与用哪种优化器无关。

The momentum, RMSProp and Adam of Lesson 6.3 are, in real training, all run on minibatch gradients instead of $\nabla f$. Momentum takes an exponential average of consecutive $\mathbf g_B$, averaging out part of the noise, but it also raises the effective learning rate to about $\eta/(1-\beta)$, and the noise ball scales with that effective rate; Adam's second moment $\mathbf s$ contains the noise variance, so on coordinates where the gradient is tiny and mostly noise it automatically shortens the step. These come back repeatedly in Part V ("Losses, generalization & regularization" and "Backpropagation"); for now remember: the noise comes from the minibatch, whichever optimizer you use.

8机器人中的它：边采数据边标定连杆长度In robotics: calibrating link lengths while data streams in

一台两连杆机械臂的连杆长度 $(l_1,l_2)$ 因为装配误差与出厂标称值有偏差。让手臂摆出一系列姿态 $\mathbf q_i$，用动捕测出手爪的真实位置 $\mathbf p_i$（带 5 cm 噪声）。正运动学 $\mathrm{FK}(\mathbf q_i;\mathbf l)=l_1\mathbf a_i+l_2\mathbf b_i$，其中 $\mathbf a_i=(\cos q_1,\sin q_1)$，$\mathbf b_i=(\cos(q_1+q_2),\sin(q_1+q_2))$，对 $\mathbf l$ 是线性的，所以代价

The link lengths $(l_1,l_2)$ of a two-link arm deviate from the factory nominal values because of assembly error. Let the arm visit a series of postures $\mathbf q_i$ and measure the gripper's true position $\mathbf p_i$ with motion capture (5 cm noise). The forward kinematics $\mathrm{FK}(\mathbf q_i;\mathbf l)=l_1\mathbf a_i+l_2\mathbf b_i$, with $\mathbf a_i=(\cos q_1,\sin q_1)$ and $\mathbf b_i=(\cos(q_1+q_2),\sin(q_1+q_2))$, is linear in $\mathbf l$, so the cost

$$ f(\mathbf l)=\frac1N\sum_{i=1}^{N}\tfrac12\big\|\,l_1\mathbf a_i+l_2\mathbf b_i-\mathbf p_i\big\|^2 $$

是凸二次函数（6.2 节：与仿射映射复合）。单个测量的梯度是 $\big(\mathbf a_i^\top\mathbf e_i,\ \mathbf b_i^\top\mathbf e_i\big)$，$\mathbf e_i$ 是该测量的预测误差。下图可以让 SGD 每次只用 $B$ 条测量，GD 每次用全部 300 条；横轴是用掉的测量条数而不是迭代次数，这才是真实世界里的成本。

is a convex quadratic (Lesson 6.2: composition with an affine map). A single measurement's gradient is $\big(\mathbf a_i^\top\mathbf e_i,\ \mathbf b_i^\top\mathbf e_i\big)$, with $\mathbf e_i$ the prediction error of that measurement. The figure lets SGD use only $B$ measurements per step and GD all 300; the horizontal axis counts measurements consumed, not iterations, which is the real-world cost.

FIG 5流式标定：SGD 对 GDStreaming calibration: SGD versus GD

左：灰色是真实连杆长度 $(1.00,\,0.80)$ 的手臂，琥珀色是当前估计的手臂（同样的关节角），小点是 300 个测得的手爪位置，带圈的是当前小批量。右：连杆长度空间 $(l_1,l_2)$ 的代价地形，紫色是 GD 路径，琥珀色是 SGD 路径，标着"最优解"的实心点是最小二乘最优解。下：到最优解的距离随"用掉的测量数"的变化（双对数）。GD 用它自己的最佳步长 $2/(\lambda_{\max}+\lambda_{\min})$，SGD 用滑块上的 $\eta$；GD 每用完 300 条测量（一整遍数据）才走一步。"窄姿态范围"时 $q_2$ 只在 0.05–0.75 rad 内变化，数据激励不足，代价非常扁；"多样姿态"时 $q_2$ 覆盖 0.05–2.8 rad。 Left: the gray arm has the true link lengths $(1.00,\,0.80)$, the amber arm the current estimate (same joint angles), the small dots are the 300 measured gripper positions and the circled ones the current minibatch. Right: the cost landscape over link lengths $(l_1,l_2)$, purple the GD path, amber the SGD path, the solid dot labelled "optimum" the least-squares optimum. Bottom: distance to the optimum against measurements consumed (log–log). GD uses its own best step $2/(\lambda_{\max}+\lambda_{\min})$ and SGD the slider's $\eta$; GD takes one step per 300 measurements (a full pass). In the "narrow poses" setting $q_2$ varies only within 0.05–0.75 rad, so the data are poorly exciting and the cost is very flat; in "diverse poses" $q_2$ covers 0.05–2.8 rad.

用"窄姿态范围"运行：在用掉 1000 条测量时，SGD 的误差已经比 GD 小得多（GD 才走了三步多）。Run with "narrow poses": by the time 1000 measurements are consumed, SGD's error is far below GD's (GD has taken barely three steps).
切换到"多样姿态"：代价变圆，两种方法都快得多，条件数从约 22 降到约 1.4。数据采得好，比算法选得好更重要。Switch to "diverse poses": the cost becomes round and both methods are much faster, the condition number dropping from about 22 to about 1.4. Collecting good data matters more than picking a clever algorithm.

真实系统里In real systems

对机器人学习，SGD 是日常：行为克隆、扩散策略、VLA 的训练都是在几十万到上亿条 (观测, 动作) 样本上用 AdamW + 小批量 + 余弦或线性衰减的学习率日程，前面还要加一小段预热（FIG 3 的最后一条曲线，Adam 的二阶矩刚启动时估计不准）。在线标定、在线系统辨识则是"流式"的：每个新测量只做一次小更新，不保存历史。标定的另一个现实问题就是 FIG 5 里的激励不足：姿态太单一时，$H$ 的条件数很大，不论 GD 还是 SGD 都慢，要靠主动设计采集姿态来解决。

For robot learning SGD is the daily bread: behaviour cloning, diffusion policies and VLAs are trained on hundreds of thousands to hundreds of millions of (observation, action) samples with AdamW, minibatches and a cosine or linear decaying learning-rate schedule, preceded by a short warm-up (the last curve of FIG 3; Adam's second-moment estimate is unreliable right after it starts). Online calibration and system identification are "streaming": every new measurement makes one small update and no history is stored. Another practical problem of calibration is the poor excitation of FIG 5: when postures are too uniform the condition number of $H$ is large and both GD and SGD crawl; the remedy is to design the data-collection postures actively.

9常见误区Common pitfalls

误区 1：用训练过程中的损失当成"真实损失"Pitfall 1: treating the running training loss as the true loss

训练日志里的损失是各个小批量上的损失，本身带噪声。曲线上下跳动不代表出了问题，也不代表过拟合；要比较两次运行，需要看平滑后的曲线或固定验证集上的损失，并多跑几个随机种子（FIG 3 就是 24 次平均）。

The loss in a training log is measured on individual minibatches and is itself noisy. A jumpy curve does not mean something broke, nor that you are overfitting; to compare two runs look at smoothed curves or the loss on a fixed validation set, and run several seeds (FIG 3 averages 24).

误区 2：把批量开到最大就一定最好Pitfall 2: the largest batch is always the best

批量越大，每步噪声越小，但每步成本也线性增加。过了某个"临界批量"之后，再增大 $B$ 只是浪费算力：梯度已经足够准，瓶颈在步数。另外，极大的批量常常泛化更差（噪声的正则化作用变弱），需要重新调学习率和日程。

A larger batch lowers the per-step noise, but the per-step cost grows linearly. Beyond some "critical batch size" a bigger $B$ just wastes compute: the gradient is accurate enough and the bottleneck is the number of steps. Very large batches also often generalize worse (the noise's regularizing effect weakens) and need the learning rate and schedule retuned.

误区 3：忘了打乱数据Pitfall 3: forgetting to shuffle the data

"无偏"要求小批量是总体的随机子集。机器人数据往往按时间顺序存放，相邻帧高度相关；若按顺序取，一个批量里全是同一个场景，梯度严重有偏，训练会在不同场景之间来回摆。务必在每个 epoch 前打乱，或在多个 episode 之间随机混合采样。

"Unbiased" requires the minibatch to be a random subset of the population. Robot data is usually stored in time order with highly correlated neighbouring frames; take it in order and one batch contains a single scene, the gradient is badly biased and training swings between scenes. Shuffle before every epoch, or sample at random across episodes.

误区 4：噪声总是坏事，或噪声总是好事Pitfall 4: noise is always bad, or noise is always good

噪声既让你停不稳（FIG 1、3），又能把你从鞍点和尖锐的谷里踢出去（FIG 4）。这是一个有时间尺度的权衡：训练早期，噪声帮你探索；晚期，要把它压下去才能收敛。因此"先大后小"的日程是几乎所有系统的标配，而不是一种技巧。

Noise stops you from settling (FIG 1, 3) and also kicks you out of saddles and sharp valleys (FIG 4). It is a trade-off with a time scale: early in training the noise helps you explore; late in training it must be suppressed for convergence. That is why "large first, small later" schedules are standard equipment in nearly every system rather than a trick.

10小练习Check yourself

1. 随机抽取（每个样本等概率）的小批量梯度 $\mathbf g_B$ 的期望是什么？1. What is the expectation of a uniformly sampled minibatch gradient $\mathbf g_B$?

每个样本梯度被抽到的概率相同，所以 $\mathbb E[\mathbf g_B]=\frac1B\sum\mathbb E[\nabla f_i]=\nabla f$，与 $B$ 无关：这就是"无偏"。$B$ 只影响方差（$\propto1/B$），不影响期望。Every per-sample gradient is equally likely, so $\mathbb E[\mathbf g_B]=\frac1B\sum\mathbb E[\nabla f_i]=\nabla f$, independent of $B$: that is unbiasedness. $B$ affects only the variance ($\propto1/B$), not the mean.

2. 把批量从 $B=16$ 增大到 $B=64$（有放回抽样），小批量梯度的标准差大约变为原来的多少？2. Increase the batch size from $B=16$ to $B=64$ (sampling with replacement). By what factor does the standard deviation of the minibatch gradient change?

方差 $\propto1/B$，标准差 $\propto1/\sqrt B$：$B$ 变成 4 倍，标准差变成 $1/\sqrt4=1/2$。想把噪声再减半，要再把批量乘 4：回报是递减的，这也是误区 2 的来源。The variance is $\propto1/B$, so the standard deviation is $\propto1/\sqrt B$: quadruple $B$ and it becomes $1/\sqrt4=1/2$. Halving the noise again costs another factor of 4 in batch size: diminishing returns, hence Pitfall 2.

3. 用常数学习率 $\eta$、固定批量 $B$ 的 SGD 优化一个凸二次函数，足够多步之后会发生什么？3. You run SGD with constant learning rate $\eta$ and fixed batch $B$ on a convex quadratic. After many steps, what happens?

步长足够小就不会发散（噪声有界）；但每步仍被随机踢开，稳态时的代价高度正比于 $\eta$、反比于 $B$（FIG 3 的虚线）。凸性只保证"没有坏的局部最小"，并不能消除小批量噪声。要收敛到精确最优点，需要 $\eta_k\to0$。A small enough step does not diverge (the noise is bounded), but every step is still randomly kicked, and the stationary cost height is proportional to $\eta$ and inversely proportional to $B$ (the dashed line of FIG 3). Convexity only rules out bad local minima; it does not remove minibatch noise. Converging to the exact optimum requires $\eta_k\to0$.

4. 以下学习率序列中，哪一个满足 Robbins–Monro 条件 $\sum\eta_k=\infty,\ \sum\eta_k^2<\infty$？4. Which learning-rate sequence satisfies the Robbins–Monro conditions $\sum\eta_k=\infty,\ \sum\eta_k^2<\infty$?

$\sum1/k$ 发散（调和级数），$\sum1/k^2=\pi^2/6$ 收敛，满足两个条件。常数 $\eta$ 的平方和发散；$1/k^2$ 与指数衰减的和本身就收敛，意味着总路程有限，可能还没走到最优点就"冻住"了。$\sum1/k$ diverges (harmonic series) and $\sum1/k^2=\pi^2/6$ converges, so both conditions hold. A constant $\eta$ has divergent $\sum\eta^2$; for $1/k^2$ and exponential decay even $\sum\eta_k$ converges, so the total path length is finite and the iterate may "freeze" before reaching the optimum.

5. FIG 5 里，为什么"窄姿态范围"的标定比"多样姿态"慢得多？5. In FIG 5, why is calibration with "narrow poses" much slower than with "diverse poses"?

两种设置的数据条数、噪声大小相同，代价也都是凸二次。区别在 $H=\frac1N\sum\begin{bmatrix}1&\cos q_2\\\cos q_2&1\end{bmatrix}$：$q_2$ 变化范围小时 $\cos q_2\approx0.9$，特征值约 1.9 和 0.09，$\kappa\approx22$；$q_2$ 变化大时 $\cos q_2$ 的平均值接近 0，特征值都接近 1。物理上：手臂总是几乎伸直的话，数据很难区分 $l_1$ 和 $l_2$ 各自是多少，只能确定它们的和。Both settings have the same number of points and noise, and both costs are convex quadratics. The difference is in $H=\frac1N\sum\begin{bmatrix}1&\cos q_2\\\cos q_2&1\end{bmatrix}$: with a small range of $q_2$, $\cos q_2\approx0.9$, the eigenvalues are about 1.9 and 0.09 and $\kappa\approx22$; with a wide range the mean of $\cos q_2$ is near 0 and both eigenvalues are near 1. Physically: if the arm is always nearly straight, the data can hardly tell $l_1$ and $l_2$ apart and mostly pin down only their sum.

11延伸阅读Further reading

I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, Ch. 8 — 小批量大小、学习率日程、动量与自适应方法在神经网络训练中的实践Minibatch size, learning-rate schedules, momentum and adaptive methods in neural-network training
L. Bottou, F. E. Curtis & J. Nocedal, Optimization Methods for Large-Scale Machine Learning, SIAM Review, 2018 — SGD 的收敛理论、噪声球与批量选择的严格综述A rigorous survey of SGD convergence theory, noise balls and batch-size choices
H. Robbins & S. Monro, A Stochastic Approximation Method, Annals of Mathematical Statistics, 1951 — 随机逼近的原始论文，Robbins–Monro 条件的出处The original stochastic-approximation paper, source of the Robbins–Monro conditions
S. Boyd & L. Vandenberghe, Convex Optimization, Ch. 9 — 确定性梯度法的收敛分析，是理解 SGD 噪声球的基础Convergence analysis of deterministic gradient methods, the baseline for understanding SGD's noise ball