Part V · 第 16 章从数据中学习 · 16.3Part V · Ch. 16 Learning from Data · 16.3

反向传播Backpropagation

训练一个有几百万参数的网络，每一步都要知道损失对每个参数的导数。反向传播只花大约两次前向计算的代价，就把这几百万个导数一次全部算出来。它就是 2.2 节的链式法则，按"从输出往回"的顺序执行，并且从不显式写出雅可比矩阵。这一节在一个机械臂的计算图上看着数值与伴随量流动，把反向传播看成向量–雅可比积，弄清它为什么便宜，学会用梯度检验确认代码没写错，再看深层网络里梯度为什么会消失或爆炸，以及初始化、归一化和残差连接怎样修好它。 Training a network with millions of parameters needs, at every step, the derivative of the loss with respect to every parameter. Backpropagation delivers all of them at once for roughly the price of two more forward passes. It is the chain rule of Lesson 2.2 executed from the output backward, without ever writing a Jacobian down. In this lesson you watch values and adjoints flow through the computational graph of a robot arm, see backprop as vector–Jacobian products, find out why it is cheap, learn to check a gradient so you can trust your code, and see why gradients vanish or explode in deep networks and how initialization, normalization and residual connections fix that.

约 45 分钟~45 min 前置：2.2 链式法则 · 3.3 雅可比 · 6.3 梯度下降家族 · 16.1 回归与分类Prereqs: 2.2 The chain rule · 3.3 The Jacobian · 6.3 Gradient descent family · 16.1 Regression & classification 6 个交互6 interactives

1为什么机器人需要它Why robots need this

一个把相机图像映射到关节指令的策略网络，参数 $\boldsymbol\theta$ 可能有几千万个。训练它，就是用梯度下降（6.3 节，配合 6.4 节的小批量）去最小化损失 $L(\boldsymbol\theta)$，所以每一步都需要梯度 $\nabla_{\boldsymbol\theta}L$：每个参数一个数，几千万个数，整个训练要算上几十万次。有限差分每个参数至少要多做一次前向计算（6.11 节算过：$d$ 维要 $2d$ 次求值），根本付不起。反向传播只需要再花大约两次前向计算的工作量，就把所有偏导一次算完。它没有新的数学：就是 2.2 节的链式法则，按聪明的顺序执行，并且从不显式地构造 3.3 节的雅可比矩阵。

机器人里到处都在用它：

训练每一个学习出来的部件：抓取成功率分类器（16.1 节）、视觉运动策略、动力学模型（24.1 节）、价值函数。它们的训练循环里，"算梯度"这一步就是反向传播。
可微的运动学与动力学：把"末端离目标多远"对关节角求导，就能用梯度下降做逆运动学；轨迹优化（6.13 节）需要动力学对状态和控制的导数。
可微渲染：12.4 节的 NeRF 与高斯泼溅，是把像素误差穿过渲染器反向传播回场景参数。
对输入求导，而不只是对参数：GAN 的生成器更新（19.2 节）、能量模型的朗之万采样（19.5 节），都要损失对网络输入的梯度，反向传播顺便就给了。

这一节依次讲：(1) 一个能看到每个节点的数值与伴随量的计算图；(2) 把反向传播看成向量–雅可比积，以及反向模式为什么只花前向计算的一个小倍数；(3) 用梯度检验确认你的反向传播没写错；(4) 深层网络里梯度为什么会消失或爆炸，四种标准的修法；(5) PyTorch 的 autograd 到底做了什么。

A policy network that maps camera images to joint commands can have tens of millions of parameters $\boldsymbol\theta$. Training it means minimizing a loss $L(\boldsymbol\theta)$ by gradient descent (Lesson 6.3, with the minibatches of Lesson 6.4), so every step needs the gradient $\nabla_{\boldsymbol\theta}L$: one number per parameter, tens of millions of numbers, hundreds of thousands of times over a training run. Finite differences need at least one extra forward pass per parameter (Lesson 6.11 counted $2d$ evaluations in $d$ dimensions), which is unaffordable. Backpropagation delivers every partial derivative at once for roughly the work of two more forward passes. There is no new mathematics in it: it is the chain rule of Lesson 2.2, executed in a clever order, without ever building the Jacobian matrices of Lesson 3.3.

Robots use it everywhere:

Training every learned component: grasp-success classifiers (Lesson 16.1), visuomotor policies, dynamics models (Lesson 24.1), value functions. In each training loop, "compute the gradient" is backprop.
Differentiable kinematics and dynamics: differentiate "how far is the tip from the target" with respect to the joint angles and gradient descent does inverse kinematics; trajectory optimization (Lesson 6.13) needs derivatives of the dynamics with respect to states and controls.
Differentiable rendering: the NeRF and Gaussian splatting of Lesson 12.4 backpropagate pixel errors through the renderer into the scene parameters.
Derivatives with respect to inputs, not only parameters: the generator update of a GAN (Lesson 19.2) and Langevin sampling from an energy model (Lesson 19.5) need the gradient of a loss with respect to a network's input, which backprop provides for free.

This lesson covers, in order: (1) a computational graph where you can see every node's value and adjoint; (2) backprop as vector–Jacobian products, and why reverse mode costs only a small multiple of a forward pass; (3) gradient checking, to confirm your backprop is right; (4) why gradients vanish or explode in deep networks, and the four standard fixes; (5) what PyTorch's autograd actually does.

2直觉：责任从后往前分摊Intuition: blame flows backward

把一段计算想成一条流水线：每个工位（节点）拿几个数，算出一个数。前向传播从输入跑到损失。现在对每个工位问一个问题：如果它的输出被轻轻推动一个很小的 $\varepsilon$，损失会动多少？这个灵敏度叫这个节点的伴随量（adjoint），沿用 2.2 节的记号 $\bar v=\partial L/\partial v$。损失自己的伴随量是 1。

要把"责任"往回传，一个工位只需要知道两件事：自己的伴随量，以及自己对每个输入的局部导数。后者只依赖它在前向传播时见过的那几个数，自己就能算。两者相乘，就是每个输入应分担的那一份；如果一个输入同时喂给好几个工位，它从各处分到的份额相加：

Think of a computation as a production line: each station (node) takes a few numbers and produces one. The forward pass runs the line from the inputs to the loss. Now ask every station one question: if its output were nudged by a tiny $\varepsilon$, how much would the loss move? That sensitivity is the node's adjoint, written $\bar v=\partial L/\partial v$ as in Lesson 2.2. The loss's own adjoint is 1.

To pass blame backward, a station needs only two things: its own adjoint, and its local derivative with respect to each input. The latter depends only on the few numbers it saw in the forward pass, so it can compute it alone. Their product is each input's share; when an input feeds several stations, the shares it receives from all of them add up:

$$ \hl{adj}{\bar v_i} \;=\; \hl{fan}{\sum_{j:\;i\to j}}\; \bar v_j\,\frac{\partial v_j}{\partial v_i}, \qquad \bar L = 1 . $$

其中 $i\to j$ 表示 $v_j$ 直接用到了 $v_i$，求和遍历所有这样的节点。2.2 节在一条标量链上一步步走过这个递推。这一节有三件新东西：(1) 真实的计算图会分叉与汇合，所以那个求和是关键；(2) 节点是向量和矩阵，"局部导数"变成一个我们从不写出来的雅可比（第 4 节）；(3) 计算图是成千上万层浮点运算，这改变了会出什么错（第 5、6 节）。

我们的第一个计算图根本不是神经网络，而是一台平面两连杆臂的正运动学（$l_1=1$ m，$l_2=0.8$ m，与 8.3 节同一台臂），后面接上"末端到目标 $\mathbf t$ 的距离平方"。这里的"参数"是关节角 $\mathbf q=(q_1,q_2)$，损失是 $L=\tfrac12\|\mathbf p(\mathbf q)-\mathbf t\|^2$。在这张图上做反向传播，得到的 $\partial L/\partial\mathbf q$ 必须等于 $J^\top\mathbf e$，即雅可比的转置乘以末端误差 $\mathbf e=\mathbf p-\mathbf t$（3.3 节的 $J$）。训练神经网络用的也正是同一个算法。

Here $i\to j$ means that $v_j$ uses $v_i$ directly, and the sum runs over all such nodes. Lesson 2.2 walked this recursion step by step on a chain of scalars. Three things are new here: (1) real graphs branch and merge, so the sum matters; (2) nodes are vectors and matrices, and the "local derivative" becomes a Jacobian that we never write down (Section 4); (3) the graph is thousands of layers of floating-point arithmetic, and that changes what can go wrong (Sections 5 and 6).

Our first graph is not a neural network at all but the forward kinematics of a planar two-link arm ($l_1=1$ m, $l_2=0.8$ m, the same arm as in Lesson 8.3), followed by the squared distance from the tip to a target $\mathbf t$. The "parameters" are the joint angles $\mathbf q=(q_1,q_2)$ and the loss is $L=\tfrac12\|\mathbf p(\mathbf q)-\mathbf t\|^2$. Backprop on this graph yields $\partial L/\partial\mathbf q$, which must equal $J^\top\mathbf e$, the Jacobian transpose times the tip error $\mathbf e=\mathbf p-\mathbf t$ (with the $J$ of Lesson 3.3). Training a neural network uses exactly the same algorithm.

3动手玩：一张会动的计算图Play: a live computational graph

上图是计算图：每个方框是一个基本运算，青色数字是前向传播算出的值，琥珀色数字是反向传播算出的伴随量 $\bar v$。按"单步"一步一步走：前 12 步是前向（从左到右），后 12 步是反向（从右到左），正在处理的边会标出紫色的局部导数。拖动机械臂（关节圆圈）或琥珀色的目标，整张图立刻重算。"梯度一步"执行 $\mathbf q\leftarrow\mathbf q-\eta\,\bar{\mathbf q}$，"持续下降"让它一直走下去。

The top panel is the computational graph: each box is one elementary operation, the cyan number is the value from the forward pass and the amber number is the adjoint $\bar v$ from the backward pass. Press "Step" to walk through it: the first 12 steps are the forward pass (left to right), the next 12 the backward pass (right to left), and the edges being processed are labelled with their local derivative in purple. Drag the arm (the joint circles) or the amber target and the whole graph recomputes at once. "Gradient step" applies $\mathbf q\leftarrow\mathbf q-\eta\,\bar{\mathbf q}$, and "Descend" keeps stepping.

FIG 1机械臂正运动学的计算图：值与伴随量The arm's forward kinematics as a computational graph: values and adjoints

计算图 · 青 = 值 v，琥珀 = 伴随 v̄GRAPH · cyan = value v, amber = adjoint v̄

机械臂：拖动关节与目标ARM · drag the joints and the target

计算图中，前向传播走到的节点显示值；反向传播时，伴随量还没收齐的节点用虚线框，收齐后变为实线框。下方：红、绿圆圈是关节 1、2，琥珀色十字是目标 $\mathbf t$，紫色箭头是误差 $\mathbf e=\mathbf p-\mathbf t$，虚线圆是可达范围 $l_1+l_2=1.8$ m。关节旁的红色、绿色弧形箭头是下降方向 $-\bar q_1$、$-\bar q_2$（长度与大小成正比）。右侧读数把反向传播得到的 $\bar{\mathbf q}$ 与直接用 3.3 节公式算出的 $J^\top\mathbf e$ 并排列出。 In the graph, nodes the forward pass has reached show their value; during the backward pass a node whose adjoint is still being collected has a dashed frame, which turns solid once it is complete. Below: the red and green circles are joints 1 and 2, the amber cross is the target $\mathbf t$, the purple arrow is the error $\mathbf e=\mathbf p-\mathbf t$, and the dashed circle is the reach $l_1+l_2=1.8$ m. The red and green curved arrows at the joints are the descent directions $-\bar q_1$, $-\bar q_2$ (length proportional to size). The readout puts the backprop result $\bar{\mathbf q}$ next to $J^\top\mathbf e$ computed directly with the formula of Lesson 3.3.

默认姿态 $\mathbf q=(0.5,1.1)$ rad、目标 $\mathbf t=(-0.6,1.2)$ m 时，$L=1.061$，反向传播给出 $\bar{\mathbf q}=(-1.793,-1.165)$，与 $J^\top\mathbf e$ 逐位相同。注意 $q_1$：它同时喂给 $c_1=\cos q_1$、$s_1=\sin q_1$ 和 $a=q_1+q_2$ 三个节点，所以它的伴随量是三份之和：$-0.697+0.069-1.165=-1.793$。漏掉任何一份，梯度就错了。

In the default pose $\mathbf q=(0.5,1.1)$ rad with target $\mathbf t=(-0.6,1.2)$ m, $L=1.061$ and backprop gives $\bar{\mathbf q}=(-1.793,-1.165)$, digit for digit the same as $J^\top\mathbf e$. Look at $q_1$: it feeds three nodes, $c_1=\cos q_1$, $s_1=\sin q_1$ and $a=q_1+q_2$, so its adjoint is the sum of three shares: $-0.697+0.069-1.165=-1.793$. Drop any one and the gradient is wrong.

试一试Try this

按"单步"（或"播放两遍传播"）从头走完前向和反向两遍传播（共 24 步）。盯住 $q_1$ 的伴随量：它被填了三次，分别来自哪三个节点？Press "Step" (or "Play both passes") and walk through the forward and backward passes from the start (24 steps). Watch $q_1$'s adjoint: it is filled in three times. Which three nodes send it a share?
从 $L>0.05$ 的位置开始，只用"梯度一步"或"持续下降"（中途不拖动）把损失降到 $10^{-4}$ 以下。从默认姿态出发、$\eta=0.3$ 时需要几步？Starting from a pose with $L>0.05$, use only "Gradient step" or "Descend" (no dragging in between) to bring the loss below $10^{-4}$. From the default pose with $\eta=0.3$, how many steps does it take?
把目标拖到虚线圆外面（至少 5 cm），再"持续下降"，直到 $\|\nabla L\|<0.002$（至少 5 步下降、中途不拖动）。梯度为零，损失却不为零：梯度为零到底保证了什么？Drag the target outside the dashed circle (by at least 5 cm), then Descend until $\|\nabla L\|<0.002$ (at least 5 descent steps, no dragging in between). The gradient is zero but the loss is not: what does a zero gradient actually promise?

从默认姿态出发，$\eta=0.3$ 走 20 步就让 $L<10^{-4}$。把 $\eta$ 调到 1 左右，下降就不再收敛，而是来回振荡（6.3 节的步长问题）。目标在可达范围外时，手臂最后伸直指向目标，$L$ 停在 $\tfrac12(\|\mathbf t\|-1.8)^2$：这是这个问题能做到的最好结果，梯度为零只说明"附近没有更好的"。目标离基座超过约 3 m 时，$\eta=0.3$ 又会来回振荡：损失的曲率随距离增大，需要把 $\eta$ 调小。

From the default pose, $\eta=0.3$ needs 20 steps to reach $L<10^{-4}$. Push $\eta$ to around 1 and the descent no longer converges but oscillates (the step-size issue of Lesson 6.3). With the target out of reach, the arm ends up straight and pointing at it, with $L$ stuck at $\tfrac12(\|\mathbf t\|-1.8)^2$: the best this problem allows, and a zero gradient only says "nothing nearby is better". With a target more than about 3 m from the base, $\eta=0.3$ oscillates again: the curvature of the loss grows with the distance, so lower $\eta$.

核心洞见Key insight

反向传播与神经网络无关：它是对任何由可微小块拼成的程序做反向模式求导。每个节点只需要自己的局部导数，图的结构负责其余一切；一个值被多处使用时，伴随量相加。在这张图上，它算出的恰好是 $J^\top\mathbf e$，而全程没有构造过 $J$。

Backprop has nothing specifically to do with neural networks: it is reverse-mode differentiation of any program built from differentiable pieces. Each node needs only its local derivative and the graph's structure does the rest; when a value is used in several places, its adjoints add. On this graph it computes exactly $J^\top\mathbf e$ without ever building $J$.

4第二、第三视角：向量–雅可比积，以及它为什么便宜Second and third views: vector–Jacobian products, and why they are cheap

反向传播从不构造雅可比，只乘它的转置Backprop never builds a Jacobian; it multiplies by its transpose

把一整层看成一个节点：$\mathbf y=f(\mathbf x)$，$\mathbf x\in\mathbb R^n$，$\mathbf y\in\mathbb R^m$，雅可比 $J=\partial\mathbf y/\partial\mathbf x$ 是 $m\times n$ 矩阵（3.3 节）。它有两种用法：

向前推一个切向量（JVP，Jacobian–vector product）：输入动一点 $\delta\mathbf x$，输出动 $\delta\mathbf y\approx J\,\delta\mathbf x$。
向后拉一个伴随量（VJP，vector–Jacobian product）：已知输出的伴随量 $\bar{\mathbf y}=\partial L/\partial\mathbf y$，输入的伴随量是 $\bar{\mathbf x}=J^\top\bar{\mathbf y}$。理由只有一行：$\delta L=\bar{\mathbf y}^\top\delta\mathbf y=\bar{\mathbf y}^\top J\,\delta\mathbf x=(J^\top\bar{\mathbf y})^\top\delta\mathbf x$。

对一层 $\mathbf y=\sigma(W\mathbf x+\mathbf b)$，记 $\mathbf z=W\mathbf x+\mathbf b$，雅可比是 $J=\mathrm{diag}(\sigma'(\mathbf z))\,W$，于是

Treat a whole layer as one node: $\mathbf y=f(\mathbf x)$ with $\mathbf x\in\mathbb R^n$, $\mathbf y\in\mathbb R^m$ and the $m\times n$ Jacobian $J=\partial\mathbf y/\partial\mathbf x$ (Lesson 3.3). It can be used in two ways:

Push a tangent forward (JVP, Jacobian–vector product): move the input by $\delta\mathbf x$ and the output moves by $\delta\mathbf y\approx J\,\delta\mathbf x$.
Pull an adjoint backward (VJP, vector–Jacobian product): given the output's adjoint $\bar{\mathbf y}=\partial L/\partial\mathbf y$, the input's adjoint is $\bar{\mathbf x}=J^\top\bar{\mathbf y}$. The reason is one line: $\delta L=\bar{\mathbf y}^\top\delta\mathbf y=\bar{\mathbf y}^\top J\,\delta\mathbf x=(J^\top\bar{\mathbf y})^\top\delta\mathbf x$.

For a layer $\mathbf y=\sigma(W\mathbf x+\mathbf b)$, write $\mathbf z=W\mathbf x+\mathbf b$; the Jacobian is $J=\mathrm{diag}(\sigma'(\mathbf z))\,W$, so

$$ \hl{jvp}{\delta\mathbf y = \sigma'(\mathbf z)\odot\big(W\,\delta\mathbf x\big)}, \qquad \hl{vjp}{\bar{\mathbf x} = W^{\top}\big(\hl{sig}{\sigma'(\mathbf z)}\odot\bar{\mathbf y}\big)} . $$

一次逐元素乘法，加一次乘 $W^\top$。宽 4096 的一层，$J$ 有一千六百多万个元素，可 VJP 的代价和前向的那次矩阵乘法一样。下图是一层 $2\to2$ 的网络（$W=\begin{bmatrix}1.3&-0.7\\0.5&1.2\end{bmatrix}$，$\mathbf b=(0.1,-0.2)$）。左边是输入平面，右边是输出平面。拖动输入点 $\mathbf x$、青色的切向量 $\delta\mathbf x$ 和右边琥珀色的伴随量 $\bar{\mathbf y}$。

One elementwise product and one multiplication by $W^\top$. A layer of width 4096 has a $J$ with over sixteen million entries, yet the VJP costs the same as the forward matrix multiply. The figure shows one $2\to2$ layer ($W=\begin{bmatrix}1.3&-0.7\\0.5&1.2\end{bmatrix}$, $\mathbf b=(0.1,-0.2)$): the input plane on the left, the output plane on the right. Drag the input point $\mathbf x$, the cyan tangent $\delta\mathbf x$, and the amber adjoint $\bar{\mathbf y}$ on the right.

FIG 2一层网络的 JVP 与 VJP：同一个雅可比的两面JVP and VJP of one layer: two faces of one Jacobian

输入平面 x · 拖动 x、δxINPUT PLANE x · drag x, δx

输出平面 y = σ(Wx+b) · 拖动 ȳOUTPUT PLANE y = σ(Wx+b) · drag ȳ

青色：切向量 $\delta\mathbf x$ 与它的像 $J\delta\mathbf x$（实线），青色虚线（末端为空心圆）是线段 $\mathbf x+s\,\delta\mathbf x$（$0\le s\le1$）的真实像，终点就是 $f(\mathbf x+\delta\mathbf x)$，$\delta\mathbf x$ 越短，它与 $J\delta\mathbf x$ 越接近。琥珀色：伴随量 $\bar{\mathbf y}$ 与拉回的 $\bar{\mathbf x}=J^\top\bar{\mathbf y}$。左图的底色是标量函数 $\bar{\mathbf y}^\top f(\mathbf x)$：紫色表示比当前点高，蓝色表示比当前点低，颜色越浓差得越多。$\bar{\mathbf x}$ 正是它的梯度，指向紫色最陡的方向。红色、绿色虚线是 $z_1=0$、$z_2=0$：单元 1、2 在这条线附近最敏感（tanh、sigmoid），或在这条线上开关（ReLU）。 Cyan: the tangent $\delta\mathbf x$ and its image $J\delta\mathbf x$ (solid); the dashed cyan curve with a hollow end is the true image of the segment $\mathbf x+s\,\delta\mathbf x$ ($0\le s\le1$), ending at $f(\mathbf x+\delta\mathbf x)$; the shorter $\delta\mathbf x$, the closer it is to $J\delta\mathbf x$. Amber: the adjoint $\bar{\mathbf y}$ and the pulled-back $\bar{\mathbf x}=J^\top\bar{\mathbf y}$. The left background is the scalar function $\bar{\mathbf y}^\top f(\mathbf x)$: purple where it is higher than at the current point, blue where lower, more colour = larger difference. $\bar{\mathbf x}$ is exactly its gradient and points up the steepest purple slope. The red and green dashed lines are $z_1=0$ and $z_2=0$: units 1 and 2 are most sensitive near them (tanh, sigmoid) or switch on and off across them (ReLU).

读数里两个内积 $\langle\bar{\mathbf y},J\delta\mathbf x\rangle$ 与 $\langle\bar{\mathbf x},\delta\mathbf x\rangle$ 永远相等（默认设置下都是 0.831）：这就是"$J^\top$ 是 $J$ 的伴随"这句话的全部含义。把 $\mathbf x$ 拖进两个单元都饱和的角落，$\sigma'(\mathbf z)$ 趋于 0，不管 $\bar{\mathbf y}$ 多大，$\bar{\mathbf x}$ 都几乎为零：梯度消失在单独一层里就已经出现了。

The two inner products in the readout, $\langle\bar{\mathbf y},J\delta\mathbf x\rangle$ and $\langle\bar{\mathbf x},\delta\mathbf x\rangle$, are always equal (0.831 for both at the default settings): that is all "$J^\top$ is the adjoint of $J$" means. Drag $\mathbf x$ into a corner where both units saturate: $\sigma'(\mathbf z)$ goes to 0 and $\bar{\mathbf x}$ nearly vanishes, however large $\bar{\mathbf y}$ is. The vanishing gradient already appears in a single layer.

试一试Try this

用 tanh，保持 $\|\bar{\mathbf y}\|\ge0.3$，把 $\mathbf x$ 拖到 $\|\bar{\mathbf x}\|/\|\bar{\mathbf y}\|<0.03$ 的地方（默认是 1.06）。此时两个单元的 $z_i$ 是多大？再切到 ReLU，在两条虚线都"关"着的那个区域里，$\bar{\mathbf x}$ 是多少？With tanh and $\|\bar{\mathbf y}\|\ge0.3$, drag $\mathbf x$ to where $\|\bar{\mathbf x}\|/\|\bar{\mathbf y}\|<0.03$ (it is 1.06 by default). How large are the two units' $z_i$ there? Then switch to ReLU: in the region where both units are off, what is $\bar{\mathbf x}$?

前向模式与反向模式：谁更便宜Forward mode and reverse mode: which is cheaper

整个网络的雅可比是各层雅可比之积 $J=J_D\cdots J_2J_1$（3.3 节的链式法则）。2.2 节说过，这个乘积可以从右往左乘，也可以从左往右乘：

前向模式：从输入端出发，每一遍推一个切向量 $\dot{\mathbf x}$ 穿过整个网络（每层一次 JVP），得到 $J$ 的一列。要整个 $m\times n$ 的 $J$，跑 $n$ 遍。
反向模式：先跑一次前向并存下每层的中间值，再从输出端拉一个伴随量 $\bar{\mathbf y}$ 回来（每层一次 VJP），得到 $J$ 的一行。要整个 $J$，跑 $m$ 遍。

训练时损失是标量，$m=1$：一遍反向就得到对所有参数的梯度。下图在一个 $n\to32\to32\to m$ 的 tanh 网络上真的把两种模式都跑一遍，逐个数乘加和逐元素运算，并检查两种方法得到的 $J$ 是否一致。

The whole network's Jacobian is the product of the layer Jacobians, $J=J_D\cdots J_2J_1$ (the chain rule of Lesson 3.3). As Lesson 2.2 said, the product can be multiplied right to left or left to right:

Forward mode: start at the input and push one tangent $\dot{\mathbf x}$ through the whole network per sweep (one JVP per layer), which yields one column of $J$. The full $m\times n$ Jacobian takes $n$ sweeps.
Reverse mode: run one forward pass that stores every layer's intermediate values, then pull one adjoint $\bar{\mathbf y}$ back from the output per sweep (one VJP per layer), which yields one row of $J$. The full Jacobian takes $m$ sweeps.

In training the loss is a scalar, $m=1$: one backward sweep gives the gradient with respect to every parameter. The figure really runs both modes on an $n\to32\to32\to m$ tanh network, counting every multiply-add and elementwise operation, and checks that the two $J$s agree.

FIG 3完整雅可比的代价：前向模式 vs 反向模式Cost of a full Jacobian: forward mode vs reverse mode

代价（单位：一次前向）vs 输入维数 nCOST (in forward passes) vs. INPUT SIZE n

前向模式forward mode反向模式reverse mode

纵轴是对数刻度，单位是"同一网络的一次普通前向计算"的运算数。青色线：前向模式，$n$ 次 JVP；琥珀色线：反向模式，一次存储中间值的前向加 $m$ 次 VJP。竖线和圆点是当前的 $n$。曲线由运算计数公式画出，读数里的数字是在当前 $n,m$ 上真正执行两种算法时数出来的，二者一致。 The vertical axis is logarithmic, in units of the operation count of one plain forward pass of the same network. Cyan: forward mode, $n$ JVPs. Amber: reverse mode, one forward pass that stores intermediates plus $m$ VJPs. The vertical line and dots mark the current $n$. The curves come from the operation-count formulas; the readout numbers are counted while both algorithms actually run at the current $n,m$, and the two agree.

试一试Try this

设 $m=1$（标量损失），$n\ge48$。反向模式比前向模式便宜多少倍？这个倍数和 $n$ 是什么关系？Set $m=1$ (a scalar loss) and $n\ge48$. How many times cheaper is reverse mode than forward mode? How does that factor relate to $n$?
找一组 $n,m$，让前向模式更便宜。比如"2 个关节角 → 末端位置与很多个特征点的像素坐标"这类少输入、多输出的函数。Find an $n,m$ where forward mode is cheaper: for example, a function with few inputs and many outputs, such as "2 joint angles → the tip position and the pixel coordinates of many feature points".

$m=1$ 时反向模式便宜 $n$ 倍左右（$n=48$ 时正好 48 倍）。前向模式的一遍 JVP 约是两次前向（原值和切向量各算一遍），反向模式的一遍 VJP 约是一次前向，所以当 $2n<m+1$ 左右时前向模式更便宜，例如 $n=2$、$m=8$：4.1 对 9.4。读数的最后一行是对全部参数求一个标量损失的梯度：反向模式要 3.0 次前向的工作量（一次前向，加上 VJP 和 $\bar W=\bar{\mathbf z}\mathbf h^\top$ 各约一次），前向模式则要对每个参数推一遍切向量。

With $m=1$ reverse mode is about $n$ times cheaper (exactly 48 times at $n=48$). One forward-mode JVP sweep costs about two forward passes (the primal and the tangent), one reverse-mode VJP sweep about one, so forward mode wins roughly when $2n<m+1$, for example $n=2$, $m=8$: 4.1 against 9.4. The readout's last line is the gradient of one scalar loss with respect to all parameters: reverse mode needs the work of 3.0 forward passes (one forward pass, plus about one each for the VJP and for $\bar W=\bar{\mathbf z}\mathbf h^\top$), while forward mode would push one tangent per parameter.

一个概念，三个视角One idea, three views

图的视角（FIG 1）：每个节点用局部导数乘自己的伴随量，分给输入，分叉处相加。线性代数视角（FIG 2）：反向传播就是用一个行向量 $\bar{\mathbf y}^\top$ 从左边去乘 $J_D\cdots J_1$，每次只做"向量乘矩阵"，所以从不需要构造任何雅可比。代价视角（FIG 3）：一个标量对任意多个输入的梯度，只花前向计算的一个小常数倍，这叫"廉价梯度原理"（Baur 与 Strassen 1983；Griewank 与 Walther 的专著）。代价是内存：反向模式必须记住前向时的中间值。

Graph view (FIG 1): each node multiplies its adjoint by its local derivatives and hands the shares to its inputs, adding at forks. Linear-algebra view (FIG 2): backprop multiplies $J_D\cdots J_1$ from the left by a row vector $\bar{\mathbf y}^\top$, always "vector times matrix", so no Jacobian is ever built. Cost view (FIG 3): the gradient of one scalar with respect to any number of inputs costs a small constant multiple of a forward pass, the "cheap gradient principle" (Baur and Strassen 1983; Griewank and Walther's book). The price is memory: reverse mode must remember the forward pass's intermediate values.

5信任但要验证：梯度检验Trust, but verify: gradient checking

手写的反向传播很容易错得很隐蔽：程序照常运行，损失甚至还在下降，只是更慢，或者停在更差的地方。验证方法是把它和数值差分逐项对比。对第 $k$ 个参数，中心差分是

Hand-written backprop is easy to get wrong in quiet ways: the program runs, the loss even goes down, only more slowly or to a worse point. The check is to compare it entry by entry with a numerical difference. For parameter $k$, the central difference is

$$ \hl{cen}{\tilde g_k=\frac{L(\boldsymbol\theta+h\mathbf e_k)-L(\boldsymbol\theta-h\mathbf e_k)}{2h}}, \qquad E=\frac{\|\mathbf g-\tilde{\mathbf g}\|}{\|\mathbf g\|+\|\tilde{\mathbf g}\|} . $$

其中 $\mathbf g$ 是反向传播给出的梯度，$\tilde{\mathbf g}$ 是数值梯度，$E$ 是相对误差（分母让它与梯度的尺度无关）。前向差分 $\hl{fdd}{(L(\boldsymbol\theta+h\mathbf e_k)-L(\boldsymbol\theta))/h}$ 少算一次，但误差大得多。下图检验一个 $[2,8,8,1]$ 的 tanh 网络（105 个参数），它在 16 个样本上拟合 $y=\cos q_1+0.5\cos(q_1+q_2)$（一个两连杆臂关节 1 重力力矩形状的函数）。表格按参数组列出相对误差；你可以往反向传播代码里植入四种常见 bug，看检验能不能抓住它们，还可以把损失的计算切到 float32。

Here $\mathbf g$ is the backprop gradient, $\tilde{\mathbf g}$ the numerical one, and $E$ the relative error (the denominator makes it independent of the gradient's scale). The forward difference $\hl{fdd}{(L(\boldsymbol\theta+h\mathbf e_k)-L(\boldsymbol\theta))/h}$ saves one evaluation but is far less accurate. The figure checks a $[2,8,8,1]$ tanh network (105 parameters) fitting $y=\cos q_1+0.5\cos(q_1+q_2)$ on 16 samples (a function shaped like joint 1's gravity torque on a two-link arm). The table lists the relative error per parameter group; you can plant four common bugs in the backprop code and see whether the check catches them, and you can switch the loss computation to float32.

FIG 4梯度检验：步长 h、浮点精度与植入的 bugGradient checking: step size h, floating-point precision and planted bugs

相对误差 vs 步长 h（双对数）RELATIVE ERROR vs. STEP h (log–log)

中心差分central前向差分forward

琥珀色：中心差分；青色：前向差分（都是对全部 105 个参数的总体相对误差）；竖线是当前的 $h$。曲线右边上升是截断误差（中心差分 $\propto h^2$，前向差分 $\propto h$），左边上升是舍入误差（$\propto\varepsilon/h$，$\varepsilon$ 是机器精度）。表格的绿点表示通过（float64 下 $<10^{-5}$，float32 下 $<10^{-3}$），红点表示 $\ge10^{-2}$，几乎一定有 bug，琥珀点介于两者之间。float32 模式只把差分用的损失按 float32 逐步舍入，反向传播仍用 float64。曲线在后台逐点计算，刚切换时会一段段长出来。 Amber: central difference; cyan: forward difference (both the overall relative error over all 105 parameters); the vertical line is the current $h$. The right-hand rise is truncation error ($\propto h^2$ for central, $\propto h$ for forward differences), the left-hand rise is round-off ($\propto\varepsilon/h$, with $\varepsilon$ the machine precision). A green dot in the table means pass ($<10^{-5}$ in float64, $<10^{-3}$ in float32), red means $\ge10^{-2}$, almost certainly a bug, and amber is in between. The float32 mode rounds every step of the loss used for the differences to float32; backprop itself stays in float64. The curves are computed point by point in the background, so right after a switch they grow in pieces.

试一试Try this

无 bug、float64、中心差分：调 $h$，让总体相对误差低于 $10^{-9}$。最佳的 $h$ 在哪里？为什么不是越小越好？No bug, float64, central difference: tune $h$ until the overall relative error is below $10^{-9}$. Where is the best $h$? Why is smaller not better?
切到 float32，先把 $h$ 降到 $10^{-6}$ 或更小，看误差变成多大；再找一个 $h$，让中心差分的误差低于 $5\times10^{-5}$。Switch to float32, first lower $h$ to $10^{-6}$ or below and see how large the error gets; then find an $h$ where the central-difference error is below $5\times10^{-5}$.

float64 下中心差分的最佳 $h$ 约是 $10^{-5.25}\approx6\times10^{-6}$，误差 $3\times10^{-11}$；前向差分的最佳 $h$ 约是 $3\times10^{-8}$，误差也只有 $2\times10^{-8}$。float32 下中心差分最好只能做到约 $2\times10^{-5}$（在 $h\approx6\times10^{-3}$），而 $h=10^{-6}$ 时误差约 10%：一个完全正确的实现会被误判成有 bug。再把四种 bug 各试一遍，看哪些组出问题：

忘了乘 $\sigma'$：前两层（W₁、b₁、W₂、b₂）全错，线性输出层（W₃、b₃）仍然对，因为它没有激活函数。
$W_2$ 用错了转置：只有 $W_2$ 下面的组（W₁、b₁）错，因为错的是从第 2 层往回传的 $\bar{\mathbf h}$。
偏置没在批上求和（只留了最后一个样本）：只有偏置组错。
损失的导数多乘了 2：所有组的误差都正好是 $|2g-g|/(2g+g)=1/3$。

出错的位置本身就是线索：检验结果告诉你 bug 在哪一层、哪一步。

In float64 the best central-difference $h$ is about $10^{-5.25}\approx6\times10^{-6}$, with error $3\times10^{-11}$; the best forward-difference $h$ is about $3\times10^{-8}$, with error only $2\times10^{-8}$. In float32 the central difference can do no better than about $2\times10^{-5}$ (at $h\approx6\times10^{-3}$), and at $h=10^{-6}$ the error is about 10%: a perfectly correct implementation would be flagged as buggy. Now try the four bugs and see which groups fail:

Forgot to multiply by $\sigma'$: the first two layers (W₁, b₁, W₂, b₂) are all wrong, while the linear output layer (W₃, b₃) is still right because it has no activation.
Wrong transpose of $W_2$: only the groups below $W_2$ (W₁, b₁) fail, since what is wrong is the $\bar{\mathbf h}$ passed back through layer 2.
Bias not summed over the batch (only the last sample kept): only the bias groups fail.
Loss derivative off by a factor of 2: every group's error is exactly $|2g-g|/(2g+g)=1/3$.

Where it fails is itself the clue: the check tells you which layer and which step holds the bug.

6很深的链：梯度消失、梯度爆炸，以及四种修法Deep chains: vanishing and exploding gradients, and four fixes

深层网络是一条很长的链。第 $l$ 层输入处的伴随量是 $\bar{\mathbf h}_l=J_l^\top J_{l+1}^\top\cdots J_{D-1}^\top\,\bar{\mathbf h}_D$：每过一层，乘一次雅可比的转置。2.2 节的 FIG 3 在标量链上见过这件事：每层的放大倍数相乘，略小于 1 就指数地消失，略大于 1 就指数地爆炸。对一层 $\mathbf h_{l+1}=\phi(W_l\mathbf h_l)$，这个倍数由三样东西决定：激活函数的斜率 $\phi'$（sigmoid 处处 $\le1/4$）、权重的尺度（初始化），以及有没有恒等通路。于是有四种标准修法：

不饱和的激活函数：ReLU 在正半轴上斜率恒为 1。
合适的初始化：Xavier/Glorot 初始化 $\mathrm{Var}(w)=2/(n+m)$（tanh，$n,m$ 为输入、输出宽度；方阵层即 $1/n$），He 初始化 $\mathrm{Var}(w)=2/n$（ReLU），让每层的信号方差大致不变（第 7 节推导）。
归一化：LayerNorm 把每个样本的 $\mathbf u=W\mathbf h$ 在特征维上标准化，BatchNorm 把每个特征在一个批上标准化，让激活函数的输入一直待在有斜率的区间里。
残差连接：$\mathbf h_{l+1}=\mathbf h_l+F(\mathbf h_l)$，雅可比变成 $I+\partial F/\partial\mathbf h$，梯度有一条直通的路（17.1 节讲 ResNet 结构本身）。

下图在一个宽 32 的深层网络上做一次前向与反向（32 个随机输入、随机目标），画出每一层的伴随量大小 $\|\bar{\mathbf h}_l\|$ 与激活值的均方根。读数里的 $\rho=\|\bar{\mathbf h}_0\|/\|\bar{\mathbf h}_D\|$ 是"到达第一层的梯度"与"最后一层的梯度"之比，绿色带是 $0.1\le\rho\le10$。权重按 $\mathcal N(0,\,g^2c/n)$ 初始化，$c=2$（ReLU，He）或 $c=1$（tanh、sigmoid，Xavier），$g$ 是增益滑块。

A deep network is a long chain. The adjoint at the input of layer $l$ is $\bar{\mathbf h}_l=J_l^\top J_{l+1}^\top\cdots J_{D-1}^\top\,\bar{\mathbf h}_D$: every layer multiplies by one more Jacobian transpose. Lesson 2.2's FIG 3 showed this on a chain of scalars: the per-layer gains multiply, and slightly below 1 they vanish exponentially, slightly above 1 they explode exponentially. For a layer $\mathbf h_{l+1}=\phi(W_l\mathbf h_l)$ the gain is set by three things: the slope of the activation $\phi'$ (sigmoid's is $\le1/4$ everywhere), the scale of the weights (the initialization), and whether there is an identity path. Hence four standard fixes:

Non-saturating activations: ReLU has slope exactly 1 on the positive side.
Proper initialization: Xavier/Glorot $\mathrm{Var}(w)=2/(n+m)$ (tanh, with input and output widths $n,m$; $1/n$ for a square layer), He $\mathrm{Var}(w)=2/n$ (ReLU), which keep each layer's signal variance roughly constant (derived in Section 7).
Normalization: LayerNorm standardizes each sample's $\mathbf u=W\mathbf h$ across features and BatchNorm standardizes each feature across a batch, keeping the activation's input where it has slope.
Residual connections: $\mathbf h_{l+1}=\mathbf h_l+F(\mathbf h_l)$ has Jacobian $I+\partial F/\partial\mathbf h$, so the gradient has a straight path through (Lesson 17.1 covers the ResNet architecture itself).

The figure runs one forward and backward pass through a deep network of width 32 (32 random inputs, random targets) and plots, for every layer, the adjoint size $\|\bar{\mathbf h}_l\|$ and the RMS of the activations. The readout's $\rho=\|\bar{\mathbf h}_0\|/\|\bar{\mathbf h}_D\|$ is "gradient reaching the first layer" over "gradient at the last layer", and the green band is $0.1\le\rho\le10$. Weights are drawn from $\mathcal N(0,\,g^2c/n)$ with $c=2$ (ReLU, He) or $c=1$ (tanh, sigmoid, Xavier), and $g$ is the gain slider.

FIG 5梯度在深层网络里怎样流动How gradients flow through a deep network

log₁₀ 大小 vs 层号 llog₁₀ SIZE vs. LAYER l

‖h̄ₗ‖ / ‖h̄_D‖激活 RMSactivation RMS

琥珀色：$\|\bar{\mathbf h}_l\|/\|\bar{\mathbf h}_D\|$，即第 $l$ 层收到的梯度相对最后一层的大小（从右往左读，就是反向传播的方向）；青色：第 $l$ 层激活值的均方根（第 0 层是输入）。绿色带是 $10^{-1}$ 到 $10^{1}$。纵轴范围随数据自动伸缩，留意刻度。 Amber: $\|\bar{\mathbf h}_l\|/\|\bar{\mathbf h}_D\|$, the gradient received by layer $l$ relative to the last layer (read it right to left, the direction of backprop). Cyan: the RMS of the activations at layer $l$ (layer 0 is the input). The green band spans $10^{-1}$ to $10^{1}$. The vertical range rescales with the data, so watch the ticks.

试一试Try this

用 sigmoid、不加残差，让 $\rho<10^{-10}$。至少要多少层？换成增益 $g=2$ 能救回来吗？With sigmoid and no residual connections, make $\rho<10^{-10}$. How many layers does it take at least? Does a gain of $g=2$ rescue it?
用 ReLU、不加残差和归一化，让梯度爆炸：$\rho>10^{4}$。增益要多大、网络要多深？With ReLU, no residual connections and no normalization, make the gradient explode: $\rho>10^{4}$. How large a gain and how deep a network does it take?
做一个至少 40 层、带残差连接的 ReLU 网络，并让 $0.2<\rho<5$。提示：只加残差还不够，看青色曲线。Build a ReLU network of at least 40 layers with residual connections and $0.2<\rho<5$. Hint: residual connections alone are not enough; look at the cyan curve.

默认设置（tanh、20 层、Xavier）下 $\rho\approx0.11$，已经有些衰减。sigmoid 的斜率不超过 1/4，这里每层大约损失 4 倍：10 层 $\rho\approx6\times10^{-7}$，17 层起跌破 $10^{-10}$，20 层 $\approx2\times10^{-13}$；把增益调到 $g=2$ 也只到 $\rho\approx10^{-8}$：更大的权重确实放大了反向的信号，但也把 sigmoid 推向饱和、斜率变小，每层的增益远小于 2 倍。ReLU 加 He 初始化（$g=1$）在 40 层时 $\rho\approx0.16$，仍在绿色带附近；增益只要偏一点，深度就会把它放大：$g=1.5$、30 层时 $\rho\approx2.6\times10^5$。只加残差的 ReLU 网络，40 层时 $\rho\approx10^9$：每一块都往"残差流"里加一份正的输出，激活值越积越大（青色线一路上升），梯度也跟着涨；再加上 LayerNorm，$\rho\approx1.3$。一个出人意料的结果：不加残差、只加 BatchNorm 的 60 层 ReLU 网络，$\rho\approx2.4\times10^5$，归一化并不自动保证梯度健康。

The default (tanh, 20 layers, Xavier) gives $\rho\approx0.11$, already some decay. Sigmoid's slope is at most 1/4 and here each layer loses about a factor of 4: $\rho\approx6\times10^{-7}$ at 10 layers, below $10^{-10}$ from 17 layers on, $\approx2\times10^{-13}$ at 20; raising the gain to $g=2$ only gets to $\rho\approx10^{-8}$: larger weights do amplify the backward signal, but they also push sigmoid toward saturation where its slope shrinks, so each layer gains far less than a factor of 2. ReLU with He initialization ($g=1$) has $\rho\approx0.16$ at 40 layers, still near the green band; but a slightly wrong gain is amplified by depth: $g=1.5$ at 30 layers gives $\rho\approx2.6\times10^5$. A ReLU network with residual connections alone has $\rho\approx10^9$ at 40 layers: each block adds a positive output to the "residual stream", the activations pile up (the cyan curve keeps climbing) and the gradient grows with them; add LayerNorm and $\rho\approx1.3$. One surprise: a 60-layer ReLU network with BatchNorm but no residual connections has $\rho\approx2.4\times10^5$, so normalization alone does not guarantee healthy gradients.

它在训练时意味着什么What it means during training

$\rho$ 只是一次前向反向的诊断。下图真的训练三个同样深、同样宽（16）的网络去拟合一个机器人信号：一个关节的重力力矩加上电机的齿槽转矩纹波，$y=\cos q+0.3\sin6q$，64 个点，Adam，学习率 $3\times10^{-3}$，全批量。三者只差结构：普通（紫）、残差（青）、残差 + LayerNorm（琥珀）。

$\rho$ is only a diagnosis of one forward and backward pass. The figure really trains three networks of equal depth and width (16) to fit a robot signal, a joint's gravity torque plus the cogging ripple of its motor, $y=\cos q+0.3\sin6q$ on 64 points, with Adam at learning rate $3\times10^{-3}$ and full batches. They differ only in structure: plain (purple), residual (cyan), residual + LayerNorm (amber).

FIG 6现场训练：普通深层网络 vs 残差 vs 残差 + LayerNormLive training: plain deep net vs residual vs residual + LayerNorm

拟合：力矩 vs 关节角 qFIT · torque vs. joint angle q

训练损失 vs 步数（对数）TRAINING LOSS vs. STEP (log)

左：虚线是目标信号，点是 64 个训练样本，三条实线是三个网络当前的预测。右：三个网络的训练损失（对数纵轴）；两条水平虚线：0.27 是"永远预测平均值"的损失，0.0225 是"学会了 $\cos q$ 却没学会纹波"的损失。读数里的 $\rho$ 是当前这一步的梯度比。每帧做的步数取决于你的电脑速度，但同样步数时的结果是确定的（固定随机种子）。图离开屏幕时训练暂停。 Left: the dashed curve is the target, the dots are the 64 training samples, and the three solid curves are the networks' current predictions. Right: the three training losses (log axis); the horizontal dashed lines mark 0.27, the loss of "always predict the mean", and 0.0225, the loss of "learned $\cos q$ but not the ripple". The readout's $\rho$ is the gradient ratio at the current step. How many steps run per frame depends on your computer, but the result at a given step count is deterministic (fixed seeds). Training pauses while the figure is off screen.

试一试Try this

用 sigmoid、至少 12 块，训练至少 500 步，直到"残差 + LayerNorm"的损失低于 0.03，而普通网络仍高于 0.2。再换成 ReLU 或 tanh 重新训练：普通网络还学不动吗？With sigmoid and at least 12 blocks, train for at least 500 steps until "residual + LayerNorm" is below 0.03 while the plain network is still above 0.2. Then switch to ReLU or tanh and retrain: is the plain network still stuck?

默认（sigmoid、16 块）下，普通网络的损失在 3000 步内一直停在 0.27，正是"预测平均值"的水平：前向信号同样被冲掉了：到最后一块时，隐藏状态在 64 个 $q$ 之间的差别只有 $10^{-11}$ 量级，输出层看到的几乎是一个常数，只能学平均值；反向时前几层几乎收不到梯度（$\rho$ 在 $10^{-10}$ 量级），而 Adam 按 $\sqrt{v}+\epsilon$ 归一化步长，这么小的梯度被 $\epsilon=10^{-8}$ 压住，参数几乎不动。残差网络在 500 步时约 0.02、残差 + LayerNorm 约 0.013，1000 步时分别约 0.019 与 0.0094。残差网络先在 0.0225 附近停留很久：它学会了平滑的 $\cos q$，纹波却还没学（约 1000 步后才开始明显往下走）；残差 + LayerNorm 更早学到一部分纹波。先学平滑部分、高频部分学得慢，就是 12.4 节提到的谱偏置。换成 ReLU（He 初始化），16 块的普通网络在 700 步左右也降到 $10^{-4}$ 以下：激活函数和初始化选对了，这个深度还用不着残差。

At the default (sigmoid, 16 blocks) the plain network's loss sits at 0.27 for all of 3000 steps, exactly the "predict the mean" level. The forward signal is washed out as well: by the last block the hidden state differs across the 64 values of $q$ only at the $10^{-11}$ level, so the output layer sees an almost constant input and can only learn the mean; backward, the first layers receive almost no gradient ($\rho$ around $10^{-10}$), and Adam normalizes its step by $\sqrt{v}+\epsilon$, so gradients this small are swamped by $\epsilon=10^{-8}$ and the parameters barely move. The residual network is near 0.02 and residual + LayerNorm near 0.013 at 500 steps, and about 0.019 and 0.0094 at 1000. The residual network lingers near 0.0225 for a long time: it has learned the smooth $\cos q$ but not yet the ripple (it only starts down clearly after about 1000 steps), while residual + LayerNorm picks up part of the ripple sooner. Learning the smooth part first and the high frequencies slowly is the spectral bias mentioned in Lesson 12.4. Switch to ReLU (He initialization) and the 16-block plain network also gets below $10^{-4}$, at around 700 steps: with the right activation and initialization, this depth does not yet need residual connections.

核心洞见Key insight

反向传播本身没有错，错的是乘积：$D$ 个雅可比相乘，任何一个系统性偏离"大小约为 1"的因素都会被指数放大。四种修法都在做同一件事：让每层雅可比的大小保持在 1 附近。初始化在开始时做到，归一化在训练中一直维持，残差连接干脆在每层的雅可比里加上一个单位阵 $I$。

Backprop itself is not at fault; the product is. Multiply $D$ Jacobians and any systematic departure from "size about 1" is amplified exponentially. All four fixes do the same thing: keep each layer's Jacobian near size 1. Initialization does it at the start, normalization keeps doing it during training, and residual connections simply add an identity $I$ to every layer's Jacobian.

7再看数学The math

提示：把鼠标悬停在公式中带底色的符号上，对应的图元素会高亮（FIG 1 的伴随量、分叉边与下降弧，FIG 2 的箭头与开关线，FIG 3、FIG 4 的曲线，FIG 5 的曲线，FIG 6 的残差网络）。Tip: hover the highlighted symbols in the formulas and the matching figure element lights up (FIG 1's adjoints, fan-out edges and descent arcs, FIG 2's arrows and switching lines, the curves of FIG 3, FIG 4 and FIG 5, and FIG 6's residual networks).

计算图与反向模式Computational graphs and reverse mode

把程序写成一列中间变量 $v_1,\dots,v_N$：前几个是输入（参数和数据），每个后面的 $v_j=\varphi_j(\dots)$ 是若干个更早变量（它的父节点，记 $i\to j$）的一个基本运算，最后一个是标量损失 $L=v_N$。按拓扑顺序算一遍是前向传播。因为 $v_i$ 只能通过直接用到它的那些 $v_j$ 影响 $L$，多元链式法则（3.1 节、3.3 节）给出

Write a program as a list of intermediate variables $v_1,\dots,v_N$: the first few are inputs (parameters and data), each later $v_j=\varphi_j(\dots)$ is one elementary operation on some earlier variables (its parents; write $i\to j$ when $v_i$ is one), and the last is the scalar loss $L=v_N$. Evaluating them in topological order is the forward pass. Since $v_i$ can affect $L$ only through the $v_j$ that use it directly, the multivariable chain rule (Lessons 3.1 and 3.3) gives

$$ \hl{adj}{\bar v_i}=\frac{\partial L}{\partial v_i}=\hl{fan}{\sum_{j:\;i\to j}}\bar v_j\,\frac{\partial\varphi_j}{\partial v_i}, \qquad \bar v_N=1, $$

按逆拓扑顺序 $j=N,N-1,\dots$ 处理，每个节点处理时它所有的"子节点"都已处理完，伴随量已经齐了。实现上更自然的写法是"推"：处理 $v_j$ 时，对它的每个父节点做 $\bar v_i\mathrel{+}=\bar v_j\,\partial\varphi_j/\partial v_i$，这正是 FIG 1 里虚线框一份份收齐的过程。对机械臂，结果是

Processing in reverse topological order $j=N,N-1,\dots$ guarantees that all of a node's "children" are done when it is reached, so its adjoint is complete. In code the natural form is "push": when processing $v_j$, do $\bar v_i\mathrel{+}=\bar v_j\,\partial\varphi_j/\partial v_i$ for each parent, exactly how FIG 1's dashed frames collect their shares one by one. For the arm the result is

$$ \bar{\mathbf q}=\hl{jte}{J(\mathbf q)^{\top}\mathbf e}, \qquad J=\begin{bmatrix}-l_1s_1-l_2s_{12} & -l_2s_{12}\\ l_1c_1+l_2c_{12} & l_2c_{12}\end{bmatrix},\qquad \mathbf e=\mathbf p(\mathbf q)-\mathbf t . $$

常用运算的 VJP 规则VJP rules for common operations

深度学习框架里的每个运算都配有一个 VJP。下表是最常用的几条（批量形式：$X$ 的每一行是一个样本，$\bar Y=\partial L/\partial Y$ 与 $Y$ 同形）：

Every operation in a deep-learning framework comes with a VJP. The most common ones (batched: each row of $X$ is one sample, and $\bar Y=\partial L/\partial Y$ has the shape of $Y$):

前向Forward	反向（VJP）Backward (VJP)	说明Note
$Y=XW^\top+\mathbf 1\mathbf b^\top$	$\bar X=\bar YW,\ \ \bar W=\bar Y^\top X,\ \ \bar{\mathbf b}=\bar Y^\top\mathbf 1$	全连接层；$\bar W$ 是每个样本 $\bar{\mathbf y}\mathbf x^\top$ 之和dense layer; $\bar W$ sums $\bar{\mathbf y}\mathbf x^\top$ over samples
$Y=\phi(Z)$	$\bar Z=\phi'(Z)\odot\bar Y$	逐元素；雅可比是对角阵，不用存elementwise; the Jacobian is diagonal and never stored
$\mathbf y=\mathbf a+\mathbf b$	$\bar{\mathbf a}=\bar{\mathbf y},\ \bar{\mathbf b}=\bar{\mathbf y}$	加法把伴随量原样复制给两边addition copies the adjoint to both sides
复制（一个值用两次）copy (a value used twice)	$\bar{\mathbf x}=\bar{\mathbf y}_1+\bar{\mathbf y}_2$	与加法互为对偶：前向复制 ⇔ 反向求和dual of addition: copy forward ⇔ sum backward
广播（偏置加到每个样本）broadcast (bias added to every sample)	对被广播的维度求和sum over the broadcast dimension	FIG 4 的"偏置没求和"就是漏了这条FIG 4's "bias not summed" bug misses this rule
$L=\tfrac12\\|\hat{\mathbf y}-\mathbf y\\|^2$	$\bar{\hat{\mathbf y}}=\hat{\mathbf y}-\mathbf y$	16.1 节的均方误差the squared error of Lesson 16.1
$L=-\log\mathrm{softmax}(\mathbf z)_c$	$\bar{\mathbf z}=\mathbf p-\mathbf e_c$	16.1 节的交叉熵；合在一起算最稳定the cross-entropy of Lesson 16.1; most stable when fused

注意加法与复制的对偶：残差连接 $\mathbf h+F(\mathbf h)$ 前向时把 $\mathbf h$ 复制给两条路再相加，反向时 $\bar{\mathbf h}$ 原样流过恒等支路，再加上 $F$ 支路送回的那份，这就是残差网络梯度通畅的代数原因。

Note the duality of addition and copying: a residual connection $\mathbf h+F(\mathbf h)$ copies $\mathbf h$ into two branches and adds them, so backward $\bar{\mathbf h}$ flows unchanged through the identity branch and receives the $F$ branch's share on top, the algebraic reason residual networks pass gradients so well.

多层感知机的反向传播Backprop for a multilayer perceptron

对 $\mathbf z_l=W_l\mathbf h_{l-1}+\mathbf b_l$，$\mathbf h_l=\phi(\mathbf z_l)$（$l=1..D$，$\mathbf h_0=\mathbf x$，$\hat{\mathbf y}=\mathbf h_D$），从 $\bar{\mathbf h}_D=\partial L/\partial\hat{\mathbf y}$ 开始，对 $l=D,\dots,1$：

For $\mathbf z_l=W_l\mathbf h_{l-1}+\mathbf b_l$, $\mathbf h_l=\phi(\mathbf z_l)$ ($l=1..D$, $\mathbf h_0=\mathbf x$, $\hat{\mathbf y}=\mathbf h_D$), start from $\bar{\mathbf h}_D=\partial L/\partial\hat{\mathbf y}$ and for $l=D,\dots,1$:

$$ \bar{\mathbf z}_l=\phi'(\mathbf z_l)\odot\bar{\mathbf h}_l,\qquad \bar W_l=\bar{\mathbf z}_l\,\mathbf h_{l-1}^{\top},\qquad \bar{\mathbf b}_l=\bar{\mathbf z}_l,\qquad \bar{\mathbf h}_{l-1}=W_l^{\top}\bar{\mathbf z}_l . $$

四行就是全部（输出层若无激活，第一行取 $\phi'=1$）。反向传播需要前向时的 $\mathbf h_{l-1}$ 和 $\mathbf z_l$（或 $\phi'(\mathbf z_l)$），所以前向必须把它们存下来。

Those four lines are all of it (for an output layer without activation, take $\phi'=1$ in the first). The backward pass needs the forward pass's $\mathbf h_{l-1}$ and $\mathbf z_l$ (or $\phi'(\mathbf z_l)$), so the forward pass must store them.

代价与内存Cost and memory

一层宽为 $n\to m$、批大小 $B$ 的全连接层，前向是一次 $Bmn$ 次乘加的矩阵乘法；反向需要 $\bar X=\bar YW$ 和 $\bar W=\bar Y^\top X$ 两次同样大小的乘法。所以反向约为前向的 2 倍，一个训练步约为 3 倍，FIG 3 数出来是 3.0。只求对输入的伴随量（一次 VJP）时，一次反向约等于一次前向；再加上对权重的梯度 $\bar W$，就是这里的 2 倍（2.2 节说的也是这个数）。这与参数个数无关，而用前向模式求同样的梯度，代价正比于参数个数。一般地，

A dense layer $n\to m$ with batch size $B$ costs one $Bmn$ multiply-add product forward; backward needs two products of the same size, $\bar X=\bar YW$ and $\bar W=\bar Y^\top X$. So backward is about twice the forward pass and a training step about three times; FIG 3 counts 3.0. The adjoints of the inputs alone (one VJP) cost about one forward pass; adding the weight gradients $\bar W$ makes it twice, the figure Lesson 2.2 also gives. This does not depend on the number of parameters, whereas forward mode would pay in proportion to it. In general,

$$ \hl{fwdm}{\mathcal C_{\rightarrow}(J)\approx n\,c_{\rightarrow}\,\mathcal C(f)}, \qquad \hl{revm}{\mathcal C_{\leftarrow}(J)\approx \big(1+m\,c_{\leftarrow}\big)\,\mathcal C(f)}, $$

其中 $\mathcal C(f)$ 是一次前向的代价，$\mathcal C_{\rightarrow}$、$\mathcal C_{\leftarrow}$ 分别是用前向模式（$n$ 遍）和反向模式（$m$ 遍）求出整个 $J$ 的代价，每遍 JVP 约 $c_{\rightarrow}\approx2$ 次前向，每遍 VJP 约 $c_{\leftarrow}\approx1$ 次（FIG 3 的网络里精确地数出来），都是与 $n$ 无关的小常数。反向模式的代价在内存：要存下前向所有中间值，内存随深度 × 宽度 × 批大小增长。梯度检查点（gradient checkpointing，Chen 等 2016）只存一部分层的激活，反向时把其余的重算一遍：大约多一次前向的计算，换来内存从 $O(D)$ 降到约 $O(\sqrt D)$。

where $\mathcal C(f)$ is the cost of one forward pass and $\mathcal C_{\rightarrow}$, $\mathcal C_{\leftarrow}$ the cost of the full $J$ by forward mode ($n$ sweeps) and by reverse mode ($m$ sweeps); one JVP sweep costs about $c_{\rightarrow}\approx2$ forward passes and one VJP sweep about $c_{\leftarrow}\approx1$ (counted exactly for the network of FIG 3), small constants independent of $n$. Reverse mode pays in memory: it must keep every intermediate of the forward pass, which grows with depth × width × batch size. Gradient checkpointing (Chen et al. 2016) stores the activations of only some layers and recomputes the rest during the backward pass: about one extra forward pass of compute, in exchange for memory dropping from $O(D)$ to about $O(\sqrt D)$.

差分的误差与最佳步长Finite-difference error and the best step

由 2.4 节的泰勒展开，前向差分的截断误差是 $\tfrac h2L''$，中心差分是 $\tfrac{h^2}{6}L'''$。另一方面，$L$ 本身带着约 $\varepsilon|L|$ 的舍入误差，除以 $h$ 后变成 $\sim\varepsilon|L|/h$。两者相加取最小（$E_1$：前向差分，$E_2$：中心差分）：

By the Taylor expansion of Lesson 2.4, the forward difference has truncation error $\tfrac h2L''$ and the central difference $\tfrac{h^2}{6}L'''$. Meanwhile $L$ itself carries a round-off error of about $\varepsilon|L|$, which becomes $\sim\varepsilon|L|/h$ after dividing by $h$. Adding the two and minimizing ($E_1$: forward difference, $E_2$: central difference):

$$ \hl{fdd}{E_1(h)\approx\tfrac h2|L''|+\frac{\varepsilon|L|}{h}}\ \Rightarrow\ h^\star\sim\sqrt\varepsilon, \qquad \hl{cen}{E_2(h)\approx\tfrac{h^2}{6}|L'''|+\frac{\varepsilon|L|}{h}}\ \Rightarrow\ h^\star\sim\varepsilon^{1/3}. $$

float64 的 $\varepsilon\approx2.2\times10^{-16}$：前向差分 $h^\star\sim1.5\times10^{-8}$，中心差分 $h^\star\sim6\times10^{-6}$；float32 的 $\varepsilon\approx1.2\times10^{-7}$：中心差分 $h^\star\sim5\times10^{-3}$。这正是 FIG 4 里曲线谷底的位置（$3\times10^{-8}$、$6\times10^{-6}$、$6\times10^{-3}$），常数因子取决于 $L$ 的各阶导数。所以梯度检验要在 float64 下做。

For float64, $\varepsilon\approx2.2\times10^{-16}$: $h^\star\sim1.5\times10^{-8}$ for forward and $h^\star\sim6\times10^{-6}$ for central differences; for float32, $\varepsilon\approx1.2\times10^{-7}$ and $h^\star\sim5\times10^{-3}$ for central differences. These are where FIG 4's curves bottom out ($3\times10^{-8}$, $6\times10^{-6}$, $6\times10^{-3}$), up to constants set by the derivatives of $L$. Hence: check gradients in float64.

初始化：让方差逐层保持不变Initialization: keep the variance constant layer by layer

设 $u_i=\sum_{j=1}^n w_{ij}h_j$，权重独立、零均值、方差 $\sigma_w^2$，且与 $\mathbf h$ 独立（4.3 节的方差运算）。则 $\mathrm{Var}(u_i)=n\,\sigma_w^2\,\mathbb E[h_j^2]$。

Let $u_i=\sum_{j=1}^n w_{ij}h_j$ with independent zero-mean weights of variance $\sigma_w^2$, independent of $\mathbf h$ (the variance rules of Lesson 4.3). Then $\mathrm{Var}(u_i)=n\,\sigma_w^2\,\mathbb E[h_j^2]$.

tanh（在原点附近近似线性）：$\mathbb E[h^2]\approx\mathrm{Var}(u)$，要方差不变就取 $\sigma_w^2=1/n$。反向传播的 $\bar{\mathbf h}=W^\top\bar{\mathbf u}$ 要求 $\sigma_w^2=1/m$（$m$ 是输出宽度）。Glorot 与 Bengio（2010）取折中 $\sigma_w^2=2/(n+m)$，即 Xavier 初始化。
tanh (roughly linear near the origin): $\mathbb E[h^2]\approx\mathrm{Var}(u)$, so constant variance needs $\sigma_w^2=1/n$. The backward pass $\bar{\mathbf h}=W^\top\bar{\mathbf u}$ wants $\sigma_w^2=1/m$ ($m$ the output width). Glorot and Bengio (2010) compromise on $\sigma_w^2=2/(n+m)$: Xavier initialization.
ReLU：对称分布的 $u$ 过 ReLU 后有一半变成 0，$\mathbb E[h^2]=\tfrac12\mathrm{Var}(u)$，所以要 $\sigma_w^2=2/n$，即 He 初始化（He 等 2015）。FIG 5 里的增益 $g$ 把它乘上 $g^2$：每层的方差乘 $g^2$，$D$ 层后乘 $g^{2D}$。
ReLU: half of a symmetric $u$ is zeroed, so $\mathbb E[h^2]=\tfrac12\mathrm{Var}(u)$ and we need $\sigma_w^2=2/n$: He initialization (He et al. 2015). The gain $g$ of FIG 5 multiplies this by $g^2$: each layer's variance is multiplied by $g^2$, and after $D$ layers by $g^{2D}$.

$$ \hl{act}{\mathrm{Var}(\mathbf h_D)}\;\approx\;\big(g^2\big)^{D}\,\mathrm{Var}(\mathbf h_0), \qquad \hl{hbar}{\|\bar{\mathbf h}_0\|}\;\approx\;\Big(\prod_{l}\big\|\mathrm{diag}(\phi'(\mathbf u_l))\,W_l\big\|\Big)\,\|\bar{\mathbf h}_D\| . $$

这只是平均意义上的结论：有限宽度时每层的增益是随机的，乘起来像一个对数正态的随机游走，所以 FIG 5 里 He 初始化的 ReLU 网络的 $\rho$ 也会在 1 附近漂移（40 层时约 0.16）。sigmoid 的 $\phi'\le1/4$，而且在零点附近 $\phi\approx\tfrac12+\tfrac u4$ 带一个 1/2 的偏移，没有一个增益能同时把前向与反向的方差都保持住。

This holds only on average: at finite width each layer's gain is random and their product behaves like a log-normal random walk, so even the He-initialized ReLU network of FIG 5 drifts away from $\rho=1$ (about 0.16 at 40 layers). Sigmoid has $\phi'\le1/4$ and near zero $\phi\approx\tfrac12+\tfrac u4$ with an offset of 1/2, so no gain keeps both the forward and the backward variance in place.

残差连接与归一化的雅可比Jacobians of residual connections and normalization

残差块 $\mathbf h_{l+1}=\mathbf h_l+F_l(\mathbf h_l)$ 的雅可比是 $I+\partial F_l/\partial\mathbf h_l$，于是

A residual block $\mathbf h_{l+1}=\mathbf h_l+F_l(\mathbf h_l)$ has Jacobian $I+\partial F_l/\partial\mathbf h_l$, so

$$ \hl{resid}{\bar{\mathbf h}_l=\bar{\mathbf h}_{l+1}+\Big(\frac{\partial F_l}{\partial\mathbf h_l}\Big)^{\!\top}\bar{\mathbf h}_{l+1}} \quad\Rightarrow\quad \bar{\mathbf h}_0=\prod_{l}\Big(I+\tfrac{\partial F_l}{\partial\mathbf h_l}\Big)^{\!\top}\bar{\mathbf h}_D . $$

即使每个 $\partial F_l/\partial\mathbf h_l$ 都很小，乘积也不会消失：展开后第一项就是 $\bar{\mathbf h}_D$ 本身。但它可能变大：如果每块都往同一个方向加东西（ReLU 的输出全为正），激活和梯度都会随深度增长，这就是 FIG 5 里"只加残差的 ReLU 网络"。

Even if every $\partial F_l/\partial\mathbf h_l$ is small the product cannot vanish: its first term is $\bar{\mathbf h}_D$ itself. It can grow, though: if every block adds something in the same direction (ReLU outputs are all positive), activations and gradients grow with depth, the "residual-only ReLU network" of FIG 5.

LayerNorm 对每个样本做 $\hat{\mathbf u}=(\mathbf u-\mu)/s$，$\mu=\tfrac1n\sum_iu_i$，$s=\sqrt{\tfrac1n\sum_i(u_i-\mu)^2+\epsilon}$。它的 VJP 是

LayerNorm maps each sample to $\hat{\mathbf u}=(\mathbf u-\mu)/s$ with $\mu=\tfrac1n\sum_iu_i$ and $s=\sqrt{\tfrac1n\sum_i(u_i-\mu)^2+\epsilon}$. Its VJP is

$$ \bar{\mathbf u}=\frac1s\Big(\bar{\hat{\mathbf u}}-\big\langle\bar{\hat{\mathbf u}}\big\rangle\,\mathbf 1-\hat{\mathbf u}\,\big\langle\bar{\hat{\mathbf u}}\odot\hat{\mathbf u}\big\rangle\Big), $$

其中 $\langle\cdot\rangle$ 是对 $n$ 个特征取平均。即把伴随量减去均值方向和 $\hat{\mathbf u}$ 方向上的分量，再除以 $s$：归一化对整体平移和缩放不敏感，反向时这两个方向的梯度就被投影掉了。BatchNorm 的公式相同，只是"均值"换成在批上对每个特征取，所以一个样本的梯度依赖于同批的其他样本。实际网络里归一化后还有可学习的缩放 $\gamma$ 和平移 $\beta$，这里为了看清梯度流省略了。

where $\langle\cdot\rangle$ averages over the $n$ features. That is, the adjoint loses its components along the mean direction and along $\hat{\mathbf u}$, then is divided by $s$: normalization is insensitive to an overall shift and scale, so backward those two directions are projected out of the gradient. BatchNorm has the same formula with the "mean" taken per feature over the batch, so one sample's gradient depends on the other samples in its batch. Real networks follow normalization with a learnable scale $\gamma$ and shift $\beta$, omitted here to keep the gradient flow clear.

8机器人中的它：autograd，以及哪里用前向、哪里用反向In robotics: autograd, and where forward or reverse mode is used

PyTorch 的 autograd 做了什么What PyTorch's autograd does

机器人学习的代码几乎都建在 PyTorch 或 JAX 上（例如 Hugging Face 的 LeRobot 用 PyTorch，MuJoCo 的 MJX 与 Brax 用 JAX）。PyTorch 的 autograd 就是本节的反向模式，外加一点工程：

边算边建图（define-by-run）：对 requires_grad=True 的张量每做一次运算，就在图里记下一个节点（grad_fn），并保存反向需要的张量。Python 的 if 和循环都照常工作，因为图就是这一次实际执行的记录。
loss.backward() 从 $\bar L=1$ 出发，按逆拓扑顺序调用每个节点的 VJP，把结果累加到叶子张量的 .grad 里。因为是累加，每一步训练前都要 optimizer.zero_grad()。
反向完成后图默认被释放（要再反向一次需 retain_graph=True）；torch.no_grad() 里不建图，推理时省内存；.detach() 把一个张量从图中切开。
自定义运算写成 torch.autograd.Function，自己给出 forward 和 backward（即 VJP），用 torch.autograd.gradcheck 以 double 精度做第 5 节的检验。
torch.func 提供 vjp、jvp、jacrev、jacfwd：前向模式与反向模式都有，按 FIG 3 的规则选。JAX 的 jax.grad、jax.vjp、jax.jvp、jacrev、jacfwd 是同样的分工。

FIG 1 那张图用 PyTorch 写只有几行，打印出的梯度就是图里的 $\bar{\mathbf q}$：

Robot-learning code is almost all built on PyTorch or JAX (Hugging Face's LeRobot uses PyTorch; MuJoCo's MJX and Brax use JAX). PyTorch's autograd is this lesson's reverse mode plus some engineering:

The graph is built while computing (define-by-run): every operation on a tensor with requires_grad=True records a node (grad_fn) and saves the tensors its backward needs. Python ifs and loops work as usual, because the graph is a record of this particular execution.
loss.backward() starts from $\bar L=1$, calls each node's VJP in reverse topological order and accumulates the results into the .grad of the leaf tensors. Because it accumulates, every training step must call optimizer.zero_grad() first.
After the backward pass the graph is freed by default (backward again needs retain_graph=True); inside torch.no_grad() no graph is built, which saves memory at inference; .detach() cuts a tensor out of the graph.
A custom operation is a torch.autograd.Function with your own forward and backward (the VJP), checked in double precision with torch.autograd.gradcheck, the test of Section 5.
torch.func offers vjp, jvp, jacrev, jacfwd: both forward and reverse mode, chosen by the rule of FIG 3. JAX's jax.grad, jax.vjp, jax.jvp, jacrev, jacfwd divide the work the same way.

FIG 1's graph takes a few lines of PyTorch, and the printed gradient is the figure's $\bar{\mathbf q}$:

import torch l1, l2 = 1.0, 0.8 q = torch.tensor([0.5, 1.1], requires_grad=True) # leaf tensor: the "parameters" t = torch.tensor([-0.6, 1.2]) # target p = torch.stack([l1*torch.cos(q[0]) + l2*torch.cos(q[0] + q[1]), l1*torch.sin(q[0]) + l2*torch.sin(q[0] + q[1])]) L = 0.5 * ((p - t)**2).sum() # forward pass builds the graph L.backward() # reverse sweep: VJPs from L back to q print(q.grad) # tensor([-1.7925, -1.1647]) = J^T e

去掉张量与 GPU，autograd 的核心不过二十行。下面是一个标量版本（与 Karpathy 的 micrograd 同一个思路）：每个节点记住自己的父节点和局部导数，backward 先做拓扑排序，再按逆序把伴随量"推"给父节点。

Strip away tensors and GPUs and the core of autograd is twenty lines. Here is a scalar version (the same idea as Karpathy's micrograd): each node remembers its parents and local derivatives, and backward sorts the graph topologically, then pushes adjoints to the parents in reverse order.

import math class Var: def __init__(self, val, parents=()): self.val, self.grad, self.parents = val, 0.0, parents # parents: [(node, local derivative)] def add(a, b): return Var(a.val + b.val, [(a, 1.0), (b, 1.0)]) def mul(a, b): return Var(a.val * b.val, [(a, b.val), (b, a.val)]) def sin(a): return Var(math.sin(a.val), [(a, math.cos(a.val))]) def backward(L): order, seen = [], set() def visit(v): # depth-first topological sort if id(v) in seen: return seen.add(id(v)) for p, _ in v.parents: visit(p) order.append(v) visit(L) L.grad = 1.0 # the loss's own adjoint for v in reversed(order): # reverse topological order for p, d in v.parents: p.grad += v.grad * d # push a share; fan-out adds up

机器人软件里的前向模式与反向模式Forward and reverse mode in robot software

真实系统里In real systems

训练网络：反向模式。抓取检测器、视觉运动策略、扩散策略、世界模型，每一步训练都是"前向 + backward() + 优化器"，损失是标量、参数上百万，正是 FIG 3 里 $m=1$ 的情形。小残差块的优化：前向模式。Ceres Solver 的 AutoDiffCostFunction 用对偶数（Jet）做前向模式自动微分：11.6 节光束法平差里的每个残差只依赖一个相机（6 维）和一个点（3 维），输入少、输出 2 维，前向模式又快又简单。Drake 用 AutoDiffXd（同样是前向模式）对动力学和约束求导。动力学的解析导数：Pinocchio 提供刚体动力学算法（RNEA、ABA）对 $\mathbf q,\dot{\mathbf q},\boldsymbol\tau$ 的解析导数（Carpentier 与 Mansard，RSS 2018），Crocoddyl 等轨迹优化与 MPC 库用它们代替通用的自动微分；CasADi 两种模式都支持，常用来写 13.4 节那样的 MPC。可微仿真：MJX、Brax 和 NVIDIA Warp 可以对仿真步求梯度，但穿过接触时梯度往往很差，这就是 6.11 节说的那种情形，也是采样方法在接触丰富的任务里仍然常用的原因之一。穿过时间的反向传播：循环网络和可微仿真沿时间展开就是一条很长的链，需要存储所有时刻的中间值，也最容易出现本节的梯度消失与爆炸（17.2 节）。

Training networks: reverse mode. Grasp detectors, visuomotor policies, diffusion policies and world models are all trained by "forward + backward() + optimizer" steps with a scalar loss and millions of parameters, the $m=1$ case of FIG 3. Optimizing small residual blocks: forward mode. Ceres Solver's AutoDiffCostFunction uses dual numbers (Jets) for forward-mode autodiff: each residual of Lesson 11.6's bundle adjustment depends on one camera (6 numbers) and one point (3), few inputs and a 2D output, where forward mode is fast and simple. Drake differentiates dynamics and constraints with AutoDiffXd, also forward mode. Analytic derivatives of dynamics: Pinocchio provides analytic derivatives of the rigid-body algorithms (RNEA, ABA) with respect to $\mathbf q,\dot{\mathbf q},\boldsymbol\tau$ (Carpentier and Mansard, RSS 2018), which trajectory-optimization and MPC libraries such as Crocoddyl use instead of generic autodiff; CasADi supports both modes and is a common way to write MPC like that of Lesson 13.4. Differentiable simulation: MJX, Brax and NVIDIA Warp can return gradients of a simulation step, but through contact those gradients are often poor, the situation described in Lesson 6.11 and one reason sampling methods remain common in contact-rich tasks. Backpropagation through time: recurrent networks and differentiable simulators unrolled over time form one very long chain that must store every time step and is the most prone to this lesson's vanishing and exploding gradients (Lesson 17.2).

9常见误区Common pitfalls

误区 1：忘了梯度是累加的Pitfall 1: forgetting that gradients accumulate

反向传播的规则是 $\bar v_i\mathrel{+}=\cdots$，框架对参数的 .grad 也是累加。不清零，第 $k$ 步用的就是前 $k$ 步梯度之和：训练会莫名其妙地发散或变慢，却不报错。反过来，这也是有意的特性：显存放不下大批量时，可以连续做几次小批量的 backward() 再走一步优化器（梯度累积），效果等于一个大批量，只要损失按小批量个数正确缩放。

Backprop's rule is $\bar v_i\mathrel{+}=\cdots$, and frameworks accumulate into parameters' .grad too. Forget to zero it and step $k$ uses the sum of the first $k$ gradients: training diverges or slows mysteriously without any error message. Conversely this is a deliberate feature: when a large batch does not fit in memory, run backward() on several small batches and then take one optimizer step (gradient accumulation), equivalent to one large batch provided the loss is scaled by the number of small batches.

误区 2：梯度检验本身做错Pitfall 2: running the gradient check wrongly

FIG 4 已经展示：float32 下用 $h=10^{-6}$，正确的代码会得到约 10% 的"误差"。所以要在 float64 下检验、用中心差分、$h$ 取 $10^{-5}$ 到 $10^{-6}$ 左右。还要小心不可导点：ReLU、max、abs 在 0 处有折角，如果 $\theta\pm h$ 跨过折角，差分会和反向传播（它只取一侧的导数）对不上；dropout 之类的随机运算要固定随机数再检验；只检验少数随机选出的参数也常常够用。

FIG 4 showed it: in float32 with $h=10^{-6}$, correct code shows about 10% "error". So check in float64, with central differences, and $h$ around $10^{-5}$ to $10^{-6}$. Beware of kinks too: ReLU, max and abs have corners at 0, and if $\theta\pm h$ straddles one, the difference will disagree with backprop (which takes a one-sided derivative); fix the random numbers of stochastic operations such as dropout before checking; and checking a few randomly chosen parameters is often enough.

误区 3：不知不觉把图切断了Pitfall 3: cutting the graph without noticing

把张量转成 Python 数（.item()）或 NumPy 数组、用 .detach() 或 .data、在 no_grad 里计算，都会让梯度在那里断掉：上游的参数收到 0 梯度，或者根本没有 .grad。离散运算（argmax、取整、对离散动作采样）的导数几乎处处为 0，梯度也传不过去，需要重参数化（19.1 节）或直通估计之类的技巧。原地修改一个反向需要的张量，PyTorch 会报 "modified by an inplace operation" 的错误，这至少是个响亮的错误。

Converting a tensor to a Python number (.item()) or a NumPy array, using .detach() or .data, or computing inside no_grad all break the gradient right there: upstream parameters receive zero gradient or no .grad at all. Discrete operations (argmax, rounding, sampling a discrete action) have zero derivative almost everywhere, so no gradient passes through them either, and you need reparameterization (Lesson 19.1) or a straight-through trick. Modifying in place a tensor that the backward pass needs makes PyTorch raise a "modified by an inplace operation" error, which is at least a loud failure.

误区 4：只盯着计算量，忘了内存Pitfall 4: watching compute and forgetting memory

反向模式便宜的代价是存储所有中间值。在高分辨率相机图像上训练、或者沿 1000 步的仿真轨迹做穿过时间的反向传播，先爆的往往是显存而不是算力。解决办法有梯度检查点（多算一次前向换内存）、截断的穿过时间反向传播、更小的批再配合梯度累积，以及混合精度。

Reverse mode is cheap in compute because it stores every intermediate value. Training on high-resolution camera images, or backpropagating through 1000 steps of a simulated trajectory, usually runs out of GPU memory before it runs out of compute. Remedies include gradient checkpointing (one extra forward pass in exchange for memory), truncated backpropagation through time, smaller batches with gradient accumulation, and mixed precision.

误区 5：部署时 BatchNorm 的行为变了Pitfall 5: BatchNorm behaving differently at deployment

训练时 BatchNorm 用当前批的均值和方差，推理时改用训练中累积的滑动平均。机器人在线控制时批大小常常是 1，如果忘了切到评估模式（model.eval()），一个样本的"批方差"为 0，输出完全错乱；即使切对了，部署环境的分布和训练时不同，滑动平均也会过时。LayerNorm 不依赖批，训练与推理行为一致，这是 Transformer 和很多机器人策略网络偏爱它的原因之一。另外，梯度爆炸时（FIG 5 的大增益），常用的保险是按全局范数做梯度裁剪（Pascanu 等 2013）。

During training BatchNorm uses the current batch's mean and variance; at inference it switches to running averages accumulated during training. Online robot control often runs with batch size 1: forget to switch to evaluation mode (model.eval()) and the "batch variance" of a single sample is 0, scrambling the output; even when switched correctly, the running averages go stale if the deployment distribution differs from training. LayerNorm does not depend on the batch and behaves identically in training and inference, one reason transformers and many robot policy networks prefer it. And when gradients explode (FIG 5's large gains), the usual safeguard is clipping the gradient by its global norm (Pascanu et al. 2013).

10小练习Check yourself

1. 节点 $u$ 同时被 $v_1=\sin u$ 和 $v_2=3u$ 使用。反向传播时已知 $\bar v_1=2$、$\bar v_2=-1$，且 $u=0$。$\bar u$ 是多少？1. Node $u$ is used by both $v_1=\sin u$ and $v_2=3u$. During backprop $\bar v_1=2$, $\bar v_2=-1$, and $u=0$. What is $\bar u$?

分叉处伴随量相加：$\bar u=\bar v_1\cos u+\bar v_2\cdot3=2\cdot1+(-1)\cdot3=-1$，选项 C。选项 A 只算了 $v_1$ 这一路，选项 B 只算了 $v_2$ 这一路，这正是 FIG 1 里 $q_1$ 的情形：漏掉一份，梯度就错。Adjoints add at a fork: $\bar u=\bar v_1\cos u+\bar v_2\cdot3=2\cdot1+(-1)\cdot3=-1$, option C. Option A counts only the $v_1$ branch and option B only the $v_2$ branch, which is FIG 1's $q_1$ situation: drop a share and the gradient is wrong.

2. 一个有一千万个参数的网络，损失是标量。用反向传播算出完整梯度，计算量大约是一次前向的多少倍？2. A network has ten million parameters and a scalar loss. Computing the full gradient by backprop costs roughly how many forward passes?

选项 B。每个全连接层的反向要两次与前向同样大小的矩阵乘法（$\bar X=\bar YW$ 与 $\bar W=\bar Y^\top X$），所以一个训练步约 3 倍前向，FIG 3 数出来是 3.0。选项 A 是前向模式或有限差分的代价。反向不是免费的，而且还要占用存储中间值的内存。Option B. Each dense layer's backward needs two products as large as the forward one ($\bar X=\bar YW$ and $\bar W=\bar Y^\top X$), so a training step costs about 3 forward passes; FIG 3 counts 3.0. Option A is the cost of forward mode or finite differences. The backward pass is not free, and it also needs memory for the stored intermediates.

3. 一层 $\mathbf y=\sigma(W\mathbf x+\mathbf b)$，已知 $\bar{\mathbf y}$。输入的伴随量 $\bar{\mathbf x}$ 是什么？3. For one layer $\mathbf y=\sigma(W\mathbf x+\mathbf b)$ with $\bar{\mathbf y}$ known, what is the input's adjoint $\bar{\mathbf x}$?

选项 B：$J=\mathrm{diag}(\sigma'(\mathbf z))W$，$\bar{\mathbf x}=J^\top\bar{\mathbf y}=W^\top(\sigma'(\mathbf z)\odot\bar{\mathbf y})$。选项 A 漏了转置，正是 FIG 4 里"用错转置"的 bug（方阵时形状对得上，不会报错）。选项 D 是 $\bar W$，不是 $\bar{\mathbf x}$。Option B: $J=\mathrm{diag}(\sigma'(\mathbf z))W$ and $\bar{\mathbf x}=J^\top\bar{\mathbf y}=W^\top(\sigma'(\mathbf z)\odot\bar{\mathbf y})$. Option A forgets the transpose, exactly FIG 4's "wrong transpose" bug (for a square matrix the shapes still match, so nothing complains). Option D is $\bar W$, not $\bar{\mathbf x}$.

4. 你在 float32 下做梯度检验，$h=10^{-7}$，所有参数组的相对误差都在 0.4 左右。最可能的原因是？4. You run a gradient check in float32 with $h=10^{-7}$, and every parameter group shows a relative error around 0.4. What is the most likely cause?

选项 C。float32 的 $\varepsilon\approx1.2\times10^{-7}$，$\varepsilon/h\approx1$：差分里几乎只剩舍入噪声。FIG 4 里 float32、$h=10^{-7}$ 时总体误差约 0.37，而换到 float64 或把 $h$ 调到 $10^{-2.5}$ 左右，同一份代码就通过了。各组误差都差不多大，也不像一个只影响某些层的 bug。Option C. Float32 has $\varepsilon\approx1.2\times10^{-7}$, so $\varepsilon/h\approx1$ and the difference is almost pure round-off noise. In FIG 4, float32 at $h=10^{-7}$ gives an overall error of about 0.37, while the same code passes in float64 or with $h$ near $10^{-2.5}$. Every group failing by a similar amount also does not look like a bug confined to some layers.

5. 一个 50 层的 sigmoid 网络，第一层收到的梯度只有最后一层的 $10^{-20}$。下面哪个改动最直接地解决问题？5. In a 50-layer sigmoid network the first layer receives $10^{-20}$ of the last layer's gradient. Which change addresses this most directly?

选项 C。恒等通路让梯度不经过 50 次 $\sigma'\le1/4$ 的衰减就能到达第一层（FIG 5：sigmoid 加残差，40 层时 $\rho\approx1$；FIG 6：残差网络能训练，普通网络停在 0.27）。选项 A 的 10 倍补不了 20 个数量级，而 Adam 这类优化器本来就按梯度大小归一化步长，太小的梯度被 $\epsilon$ 压住。选项 B 针对的是梯度爆炸。选项 D 只是让那个微小的数更精确。换成 ReLU 加 He 初始化也是一种有效的修法，但不在选项里。Option C. The identity path lets the gradient reach the first layer without 50 factors of $\sigma'\le1/4$ (FIG 5: sigmoid with residual connections has $\rho\approx1$ at 40 layers; FIG 6: the residual network trains while the plain one sits at 0.27). Option A's factor of 10 cannot make up 20 orders of magnitude, and optimizers like Adam already normalize step sizes by the gradient's size, so very small gradients are swamped by $\epsilon$. Option B addresses exploding gradients. Option D only makes that tiny number more precise. Switching to ReLU with He initialization is another valid fix, but it is not offered.

11延伸阅读Further reading

D. E. Rumelhart, G. E. Hinton & R. J. Williams, Learning Representations by Back-Propagating Errors, Nature 323, 1986 — 让反向传播在神经网络里广为人知的论文The paper that made backpropagation widely known for neural networks
S. Linnainmaa, Taylor Expansion of the Accumulated Rounding Error, BIT 16, 1976（基于他 1970 年的硕士论文based on his 1970 master's thesis）; P. J. Werbos, Beyond Regression, PhD thesis, Harvard, 1974 — 反向模式自动微分与用它训练网络的早期来源Early sources of reverse-mode differentiation and of using it to train networks
A. Griewank & A. Walther, Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, 2nd ed., SIAM, 2008 — 自动微分的标准教材：前向 / 反向模式、代价界、检查点The standard text on automatic differentiation: forward and reverse modes, cost bounds, checkpointing
W. Baur & V. Strassen, The Complexity of Partial Derivatives, Theoretical Computer Science 22, 1983 — "梯度只比函数贵一个常数倍"的经典结果The classic result that a gradient costs only a constant factor more than the function
A. G. Baydin, B. A. Pearlmutter, A. A. Radul & J. M. Siskind, Automatic Differentiation in Machine Learning: a Survey, JMLR 18, 2018 — 面向机器学习的自动微分综述A survey of automatic differentiation for machine learning
I. Goodfellow, Y. Bengio & A. Courville, Deep Learning, MIT Press, 2016, Ch. 6.5 and 8 — 反向传播与计算图；深层网络的优化难点Backpropagation and computational graphs; the difficulties of optimizing deep networks
Y. Bengio, P. Simard & P. Frasconi, Learning Long-Term Dependencies with Gradient Descent is Difficult, IEEE Trans. Neural Networks 5(2), 1994; R. Pascanu, T. Mikolov & Y. Bengio, On the Difficulty of Training Recurrent Neural Networks, ICML 2013 — 梯度消失与爆炸的分析，梯度裁剪Analysis of vanishing and exploding gradients; gradient clipping
X. Glorot & Y. Bengio, Understanding the Difficulty of Training Deep Feedforward Neural Networks, AISTATS 2010; K. He, X. Zhang, S. Ren & J. Sun, Delving Deep into Rectifiers, ICCV 2015 — Xavier 与 He 初始化Xavier and He initialization
S. Ioffe & C. Szegedy, Batch Normalization, ICML 2015; J. L. Ba, J. R. Kiros & G. E. Hinton, Layer Normalization, arXiv 2016 — 两种归一化The two normalizations
K. He, X. Zhang, S. Ren & J. Sun, Deep Residual Learning for Image Recognition, CVPR 2016 — 残差连接Residual connections
T. Chen, B. Xu, C. Zhang & C. Guestrin, Training Deep Nets with Sublinear Memory Cost, arXiv 2016 — 梯度检查点Gradient checkpointing
A. Paszke et al., PyTorch: An Imperative Style, High-Performance Deep Learning Library, NeurIPS 2019 — define-by-run 的 autograd 设计The design of define-by-run autograd
J. Carpentier & N. Mansard, Analytical Derivatives of Rigid Body Dynamics Algorithms, RSS 2018 — Pinocchio 中动力学的解析导数Analytic derivatives of the dynamics in Pinocchio
A. Karpathy, micrograd — 约一百行的标量 autograd，第 8 节代码的同一思路A scalar autograd in about a hundred lines, the same idea as the code in Section 8

前向Forward	反向（VJP）Backward (VJP)	说明Note
\(Y=XW^\top+\mathbf 1\mathbf b^\top\)	\(\bar X=\bar YW,\ \ \bar W=\bar Y^\top X,\ \ \bar{\mathbf b}=\bar Y^\top\mathbf 1\)	全连接层；\(\bar W\) 是每个样本 \(\bar{\mathbf y}\mathbf x^\top\) 之和dense layer; \(\bar W\) sums \(\bar{\mathbf y}\mathbf x^\top\) over samples
\(Y=\phi(Z)\)	\(\bar Z=\phi'(Z)\odot\bar Y\)	逐元素；雅可比是对角阵，不用存elementwise; the Jacobian is diagonal and never stored
\(\mathbf y=\mathbf a+\mathbf b\)	\(\bar{\mathbf a}=\bar{\mathbf y},\ \bar{\mathbf b}=\bar{\mathbf y}\)	加法把伴随量原样复制给两边addition copies the adjoint to both sides
复制（一个值用两次）copy (a value used twice)	\(\bar{\mathbf x}=\bar{\mathbf y}_1+\bar{\mathbf y}_2\)	与加法互为对偶：前向复制 ⇔ 反向求和dual of addition: copy forward ⇔ sum backward
广播（偏置加到每个样本）broadcast (bias added to every sample)	对被广播的维度求和sum over the broadcast dimension	FIG 4 的"偏置没求和"就是漏了这条FIG 4's "bias not summed" bug misses this rule
\(L=\tfrac12\\|\hat{\mathbf y}-\mathbf y\\|^2\)	\(\bar{\hat{\mathbf y}}=\hat{\mathbf y}-\mathbf y\)	16.1 节的均方误差the squared error of Lesson 16.1
\(L=-\log\mathrm{softmax}(\mathbf z)_c\)	\(\bar{\mathbf z}=\mathbf p-\mathbf e_c\)	16.1 节的交叉熵；合在一起算最稳定the cross-entropy of Lesson 16.1; most stable when fused