生成对抗网络 (GAN)Generative Adversarial Networks

机器人学习里有大量“一对多”的问题。同一个杯子可以从左边抓，也可以从右边抓；同一个障碍物可以从左绕，也可以从右绕；仿真里的同一张图像，换到真实相机下可以有无数种光照和纹理。用均方误差去回归这类目标，网络会学到各个答案的平均：两种抓法的平均是抓在杯把和杯身之间的空气里，两条绕行路线的平均正好撞上障碍物。我们需要的不是平均值，而是一个能采样的模型：每次给出一个合理的、属于某一种模式的答案。这就是第 19 章的生成模型要做的事。

19.1 节的 VAE 走的是似然路线：写出一个潜变量模型，最大化对数似然的下界 ELBO。生成对抗网络（GAN，Goodfellow 等，2014）换了一条完全不同的路：不写密度，只要求“骗得过一个裁判”。生成器 \(G\) 把噪声 \(\mathbf z\) 映射成样本 \(G(\mathbf z)\)；判别器 \(D\) 是一个二分类器，输出“这个样本来自真实数据”的概率；两者对着同一个目标一个最大化、一个最小化。训练好的生成器只需一次前向传播就能出一个样本，样本往往很清晰（VAE 的样本常常偏模糊），代价是训练很不稳定。理解它为什么不稳定，以及每一个“训练技巧”背后的数学，是本节的主线。

它和机器人的关系不止于生成图片。判别器“区分专家与非专家”的能力本身就很有用：GAIL 把它当作奖励来做模仿学习（20.6 节），AMP 用它让腿式机器人和仿真角色学出自然的动作风格；在 sim-to-real 里，GAN 被用来把仿真图像“翻译”成看起来像真实相机拍的图像。

Robot learning is full of one-to-many problems. The same mug can be grasped from the left or from the right; the same obstacle can be passed on either side; the same simulated image could look like countless real camera images under different lighting and textures. Regress such a target with mean squared error and the network learns the average of the answers: the average of two grasps is a grasp in the air between the handle and the body, and the average of two detours drives straight into the obstacle. What we need is not the mean but a model we can sample from, one that returns a plausible answer belonging to one of the modes each time. That is the job of the generative models of Chapter 19.

The VAE of Lesson 19.1 takes the likelihood route: write a latent-variable model and maximize a lower bound on the log-likelihood, the ELBO. Generative adversarial networks (GANs; Goodfellow et al., 2014) take an entirely different road: write no density, just "fool a referee". A generator \(G\) maps noise \(\mathbf z\) to a sample \(G(\mathbf z)\); a discriminator \(D\) is a binary classifier that outputs the probability that a sample came from the real data; the two push the same objective in opposite directions, one maximizing and one minimizing it. A trained generator produces a sample in a single forward pass and the samples are often sharp (VAE samples are often blurry), but training is notoriously unstable. Understanding why, and the mathematics behind each "training trick", is the thread of this lesson.

The connection to robots goes beyond making pictures. The discriminator's ability to "tell the expert from the non-expert" is useful by itself: GAIL uses it as a reward for imitation learning (Lesson 20.6), and AMP uses it to give legged robots and simulated characters a natural motion style; in sim-to-real transfer, GANs have been used to "translate" simulated images into images that look as if they came from a real camera.

想象一个伪造者（生成器）和一个鉴定师（判别器）。鉴定师看一堆真品和一堆伪造品，学会给每件东西打一个“是真品”的概率 \(D(x)\)。伪造者看不到真品，只能看到鉴定师的打分，并朝着“让分数变高”的方向改进伪造手法。如果鉴定师已经做到最好，他在 \(x\) 处给出的分数应该是多少？在 \(x\) 附近，真品出现的密度是 \(\hl{pdata}{p_{\text{data}}(x)}\)，伪造品出现的密度是 \(\hl{pg}{p_g(x)}\)，那么一件落在 \(x\) 的东西是真品的概率就是两者之比：

Picture a forger (the generator) and an inspector (the discriminator). The inspector looks at a pile of genuine items and a pile of forgeries and learns to give each item a probability \(D(x)\) of being genuine. The forger never sees a genuine item, only the inspector's scores, and changes the forging technique in whatever direction raises the score. If the inspector is as good as possible, what score should they give at \(x\)? Near \(x\), genuine items occur with density \(\hl{pdata}{p_{\text{data}}(x)}\) and forgeries with density \(\hl{pg}{p_g(x)}\), so the probability that an item found at \(x\) is genuine is the ratio

（两堆一样多时；第 6 节会从损失函数严格推出它。）这一个式子已经说明了 GAN 的很多性质。伪造品与真品完全一样时，\(D^*\equiv\tfrac12\)，鉴定师只能瞎猜，这正是训练的终点。伪造品离真品很远、两堆不重叠时，\(D^*\) 在真品上是 1、在伪造品上是 0，鉴定师“完美”，可伪造品附近的 \(D^*\) 是一片平坦的 0：伪造者顺着分数往上爬，却没有坡可爬。下图让你亲手摆放生成器的分布 \(p_g\)，看最优判别器怎样随之变化，并读出两者之间的 Jensen-Shannon 散度（JS）：它就是“最优鉴定师能把真假分得多开”的度量，定义和性质见第 6 节。

(when the two piles are the same size; Section 6 derives it properly from the loss.) This one formula already explains a lot about GANs. When forgeries are exactly like the genuine items, \(D^*\equiv\tfrac12\) and the inspector can only guess: that is where training should end. When forgeries are far from the genuine items and the piles do not overlap, \(D^*\) is 1 on the real items and 0 on the forgeries: the inspector is "perfect", yet around the forgeries \(D^*\) is a flat 0, and the forger, climbing the score, has no slope to climb. In the figure below you place the generator's distribution \(p_g\) yourself, watch the optimal discriminator follow, and read off the Jensen-Shannon divergence (JS) between the two: it measures "how well the best possible inspector can separate real from fake"; its definition and properties are in Section 6.

FIG 1 里的 \(D^*\) 是用真实密度算出来的；真实的 GAN 两边都不知道密度，只有样本。下图是一个真正在训练的 GAN：生成器 \(G\) 是一个 2→24→24→2 的 MLP（leaky ReLU），把二维高斯噪声 \(\mathbf z\) 变成平面上的一个点；判别器 \(D\) 是 2→24→24→1 的 MLP，输出一个 logit \(l(x)\)，\(D(x)=\sigma(l(x))\)。每次迭代：

1. 判别器一步：取 32 个真样本和 32 个 \(G(\mathbf z)\)，用二分类交叉熵（16.1 节）把真样本推向 1、假样本推向 0，Adam 走一步。
2. 生成器一步：再取 32 个 \(\mathbf z\)，把 \(G(\mathbf z)\) 送进判别器，判别器参数不动，只把损失对判别器输入的梯度 \(\partial L/\partial x\) 反向传回生成器（16.3 节的反向传播：一路穿过 \(D\)，再穿过 \(G\)），更新 \(G\) 的参数。

默认用“非饱和”生成器损失 \(-\log D(G(\mathbf z))\)，第 4 节解释为什么不用原始的那个。背景颜色是当前判别器的判断：越绿越像“真”，越红越像“假”；青色箭头是判别器给每个生成样本的“推力”\(-\partial L_G/\partial x\)，也就是 \(\hl{gradx}{\nabla_x\log D(x)}\) 的方向。

The \(D^*\) of FIG 1 was computed from the true densities; a real GAN knows neither density, only samples. The figure below is a GAN that genuinely trains: the generator \(G\) is a 2→24→24→2 MLP (leaky ReLU) that turns 2D Gaussian noise \(\mathbf z\) into a point in the plane; the discriminator \(D\) is a 2→24→24→1 MLP that outputs a logit \(l(x)\), with \(D(x)=\sigma(l(x))\). Each iteration:

1. One discriminator step: take 32 real samples and 32 samples \(G(\mathbf z)\), push real towards 1 and fake towards 0 with binary cross-entropy (Lesson 16.1), one Adam step.
2. One generator step: draw 32 fresh \(\mathbf z\), feed \(G(\mathbf z)\) into the discriminator, keep the discriminator's parameters fixed, and backpropagate the gradient of the loss with respect to the discriminator's input, \(\partial L/\partial x\), into the generator (the backpropagation of Lesson 16.3: through \(D\), then through \(G\)), updating \(G\)'s parameters.

By default the generator uses the "non-saturating" loss \(-\log D(G(\mathbf z))\); Section 4 explains why not the original one. The background shows the current discriminator's verdict: more green = "real", more red = "fake". The cyan arrows are the "push" \(-\partial L_G/\partial x\) the discriminator gives each generated sample, which points along \(\hl{gradx}{\nabla_x\log D(x)}\).

Goodfellow 等的原始目标里，生成器最小化 \(\hl{mm}{\log(1-D(G(\mathbf z)))}\)。问题出在训练初期：生成样本很差，判别器很容易认出它们，\(D(G(\mathbf z))\approx0\)。此时 \(\log(1-D)\) 对 logit 的导数是 \(-D\approx0\)：判别器越有把握，生成器的梯度越小，就在它最需要学习的时候。原论文已经给出了补救：改成最小化 \(\hl{ns}{-\log D(G(\mathbf z))}\)，对 logit 的导数是 \(-(1-D)\approx-1\)。两个损失的不动点相同（都希望 \(D(G(\mathbf z))\) 变大），但信号的强弱正好颠倒：原始损失按 \(D\) 加权每个样本，非饱和损失按 \(1-D\) 加权。第三种做法是 WGAN（Arjovsky 等，2017）：判别器不再输出概率，而是一个受 1-Lipschitz 约束的“评论家”\(f\)，生成器最小化 \(\hl{wgan}{-f(G(\mathbf z))}\)。

下图把三者放在一个能精确求解的例子里比较：真数据是 \(\mathcal N(1.5,\,0.3^2)\)，生成器是 \(\mathcal N(\theta,\,0.3^2)\)，只有一个参数 \(\theta\)。两个等宽高斯之间的最优判别器 logit 恰好是 \(x\) 的线性函数，所以可以把“判别器强度”\(\beta\) 写成 logit 的倍数：\(\beta=1\) 是最优判别器，\(\beta\) 小一些是“没练好、拿不准”的判别器。对每个 \(\theta\)，图中给出生成器损失对 \(\theta\) 的梯度大小（判别器固定）。

In the original objective of Goodfellow et al. the generator minimizes \(\hl{mm}{\log(1-D(G(\mathbf z)))}\). The trouble is early training: generated samples are poor, the discriminator spots them easily, and \(D(G(\mathbf z))\approx0\). The derivative of \(\log(1-D)\) with respect to the logit is then \(-D\approx0\): the more confident the discriminator, the smaller the generator's gradient, right when it most needs to learn. The original paper already gave the remedy: minimize \(\hl{ns}{-\log D(G(\mathbf z))}\) instead, whose derivative with respect to the logit is \(-(1-D)\approx-1\). Both losses have the same fixed point (both want \(D(G(\mathbf z))\) to rise), but the signal strength is exactly reversed: the original loss weights each sample by \(D\), the non-saturating one by \(1-D\). A third option is the WGAN (Arjovsky et al., 2017): the discriminator no longer outputs a probability but is a 1-Lipschitz "critic" \(f\), and the generator minimizes \(\hl{wgan}{-f(G(\mathbf z))}\).

The figure compares the three on an example that can be solved exactly: the real data are \(\mathcal N(1.5,\,0.3^2)\) and the generator is \(\mathcal N(\theta,\,0.3^2)\), with a single parameter \(\theta\). Between two Gaussians of equal width the optimal discriminator's logit is exactly linear in \(x\), so "discriminator strength" \(\beta\) can be written as a multiple of that logit: \(\beta=1\) is the optimal discriminator, a smaller \(\beta\) is an "undertrained, unsure" one. For every \(\theta\) the figure shows the size of the gradient of the generator loss with respect to \(\theta\) (discriminator held fixed).

读数里还给出 JS 和 Wasserstein 距离 \(W=|1.5-\theta|\)。随着 \(\theta\) 远离，JS 趋于 \(\log2\) 后就几乎不再变化，它对 \(\theta\) 的导数趋于 0；\(W\) 却始终以斜率 1 增长。这就是 WGAN 的出发点：在两个分布几乎不重叠时（高维里的常态，真实图像只占像素空间里极薄的一层），JS 失去了“还差多远”的信息，而 Wasserstein 距离保留了它。注意非饱和损失的梯度也不是“越大越好”：它随距离线性增长，判别器很尖锐时会给出很大、很吵的梯度，这是 FIG 2 里有时需要较小学习率的原因之一。

The readout also gives JS and the Wasserstein distance \(W=|1.5-\theta|\). As \(\theta\) moves away, JS approaches \(\log2\) and then hardly changes, its derivative with respect to \(\theta\) going to 0, whereas \(W\) keeps growing with slope 1. That is the starting point of the WGAN: when two distributions barely overlap (the normal situation in high dimensions, where real images occupy a very thin sliver of pixel space), JS loses the information "how far is there still to go" and the Wasserstein distance keeps it. Note that the non-saturating gradient is not "the bigger the better" either: it grows linearly with the distance, so a very sharp discriminator gives large, noisy gradients, one reason FIG 2 sometimes wants a smaller learning rate.

GAN 最出名的失败是模式坍缩：生成器只产出数据里的一种或几种模式，例如只会从一个方向去抓杯子。原因藏在目标的顺序里。理论上我们解的是 \(\min_G\max_D V\)：对每个 \(G\)，判别器先做到最好，生成器再选最好的 \(G\)。可是交替梯度下降里，生成器每一步面对的是一个固定的判别器。而对一个固定的 \(D\)，生成器的最优回应是什么？它的损失 \(-\mathbb E_{x\sim p_g}[\log D(x)]\) 对 \(p_g\) 是线性的，线性函数在概率单纯形上的最小值落在顶点：把所有质量放在 \(D\) 最大的那一点。判别器随后发现“这一点全是假货”，把那里的 \(D\) 压低，生成器又整体跳到下一个 \(D\) 最高的地方……这就是模式跳跃。

下图把这个机制剥离出来，用一个可以精确计算的离散版本：抓取器有 8 个接近方向，示教数据 \(p\) 均匀分布在这 8 个方向上；生成器是 8 个方向上的分布 \(q=\operatorname{softmax}(\boldsymbol\theta)\)；判别器每隔 \(m\) 步重新算成当前 \(q\) 下的最优判别器 \(D_k=p_k/(p_k+q_k)\)，其余时间保持不动；生成器在两次刷新之间用学习率 \(\eta\) 对非饱和损失做梯度下降。

The most famous failure of GANs is mode collapse: the generator produces only one or a few of the modes in the data, for example it only ever grasps the mug from one direction. The cause is hidden in the order of the objective. In theory we solve \(\min_G\max_D V\): for each \(G\) the discriminator does its best first, then the generator picks the best \(G\). But in alternating gradient descent, each generator step faces a fixed discriminator. And what is the generator's best response to a fixed \(D\)? Its loss \(-\mathbb E_{x\sim p_g}[\log D(x)]\) is linear in \(p_g\), and a linear function on the probability simplex is minimized at a vertex: put all the mass on the point where \(D\) is largest. The discriminator then discovers that "this point is all fakes", lowers \(D\) there, and the generator jumps wholesale to the next highest point of \(D\) … this is mode hopping.

The figure isolates this mechanism in a discrete version that can be computed exactly: a gripper has 8 approach directions and the demonstrations \(p\) are spread evenly over them; the generator is a distribution over the 8 directions, \(q=\operatorname{softmax}(\boldsymbol\theta)\); every \(m\) steps the discriminator is reset to the optimal discriminator for the current \(q\), \(D_k=p_k/(p_k+q_k)\), and stays frozen in between; between refreshes the generator does gradient descent on the non-saturating loss with learning rate \(\eta\).

真实的 GAN 介于这几种情形之间：判别器只走几步、永远“半过时”，生成器每一步又会同时移动所有样本（它是一个共享参数的函数，FIG 4 里的 softmax 也一样）。所以坍缩是否发生取决于 \(\eta\) 与 \(m\) 的乘积这类“生成器在判别器跟上之前能走多远”的量：试试把 η 调到 1，再选“每 30 步”，它会收敛。对策都围绕这个机制：让判别器“看见”一整批样本的多样性（minibatch discrimination，Salimans 等，2016）、让生成器预判判别器的回应（unrolled GAN，Metz 等，2017）、给判别器加梯度惩罚或谱归一化让它的回应更平滑，或者干脆换一个按似然训练、天然要覆盖所有数据的模型（VAE、flow、扩散模型）。

A real GAN sits between these cases: the discriminator only takes a few steps and is always "half stale", while each generator step moves all its samples together (it is a function with shared parameters, as is the softmax in FIG 4). So whether collapse happens depends on quantities like the product of \(\eta\) and \(m\), "how far the generator can travel before the discriminator catches up": try η = 1 with "every 30 steps" and it converges. The remedies all target this mechanism: let the discriminator "see" the diversity of a whole batch (minibatch discrimination; Salimans et al., 2016), let the generator anticipate the discriminator's response (unrolled GANs; Metz et al., 2017), give the discriminator a gradient penalty or spectral normalization so its responses are smoother, or switch to a model trained by likelihood, which is naturally forced to cover all the data (VAEs, flows, diffusion models).

6.3 节的梯度下降有一个总是在下降的目标函数，学习率够小就能收敛。GAN 不是这样：两个玩家沿着同一个函数的相反方向走，联合的“梯度场”可以绕圈，而不是指向一个谷底。Mescheder、Geiger 与 Nowozin（2018）用一个最小的例子把这一点说清楚了，称为 Dirac GAN：真数据是 \(x=0\) 处的一个点，生成器是 \(x=\theta\) 处的一个点，判别器是 \(D(x)=\sigma(\psi x)\)，只有一个参数 \(\psi\)。唯一的平衡点是 \(\theta=0,\ \psi=0\)。在平衡点附近，两人的更新近似为

The gradient descent of Lesson 6.3 has an objective that always decreases, and with a small enough step it converges. A GAN is different: the two players move in opposite directions on the same function, and their joint "gradient field" can circle around instead of pointing into a valley. Mescheder, Geiger and Nowozin (2018) made this precise with a minimal example, the Dirac GAN: the real data is a single point at \(x=0\), the generator is a single point at \(x=\theta\), and the discriminator is \(D(x)=\sigma(\psi x)\) with one parameter \(\psi\). The only equilibrium is \(\theta=0,\ \psi=0\). Near it, the two players' updates are approximately

（先看 \(\gamma=0\)）。这是一个纯旋转：判别器看到生成器在右边（\(\theta>0\)），就让 \(\psi\) 变负，把右边判成“假”；生成器于是往左走；走过头之后判别器又反过来……线性化矩阵的特征值是 \(\pm\tfrac i2\)，纯虚数，没有任何衰减。下图在 \((\theta,\psi)\) 平面上运行真实（非线性）的离散更新，用的是原始的零和损失（非饱和损失在平衡点处的线性化完全相同）。

(take \(\gamma=0\) for now). This is a pure rotation: the discriminator sees the generator on the right (\(\theta>0\)) and makes \(\psi\) negative, calling the right side "fake"; the generator then moves left; it overshoots and the discriminator turns around … The eigenvalues of the linearization are \(\pm\tfrac i2\), purely imaginary, with no damping at all. The figure runs the true (nonlinear) discrete updates in the \((\theta,\psi)\) plane, using the original zero-sum loss (the non-saturating loss has exactly the same linearization at the equilibrium).

“交替更新”（判别器先走一步，生成器用更新后的判别器再走一步）也不收敛，只是不再发散：在 η = 0.1 下跑 6000 步，距离始终在起点的 0.99 到 1.02 倍之间，永远绕圈。\(\hl{r1}{\gamma\psi}\) 这一项来自 R1 正则：给判别器加惩罚 \(\tfrac\gamma2\,\mathbb E_{x\sim p_{\text{data}}}\|\nabla_x l(x)\|^2\)，即判别器 logit 在真数据上的梯度平方。在 Dirac GAN 里 \(\nabla_xl=\psi\)，惩罚就是 \(\tfrac\gamma2\psi^2\)，它给旋转加上了阻尼。直觉上，R1 告诉判别器：“在真数据附近别太陡”，到了平衡点，判别器就没有理由偏离 \(\psi=0\)。γ 也不是越大越好：γ = 1 时约 140 步收敛到 2% 以内，γ = 2 反而要约 310 步（过阻尼）。WGAN-GP 的梯度惩罚、谱归一化都属于同一类“约束判别器斜率”的方法。

"Alternating" updates (the discriminator steps first, then the generator steps against the updated discriminator) do not converge either, they just stop diverging: run 6000 steps at η = 0.1 and the distance stays between 0.99 and 1.02 times the start, circling forever. The term \(\hl{r1}{\gamma\psi}\) comes from the R1 regularizer: penalize the discriminator by \(\tfrac\gamma2\,\mathbb E_{x\sim p_{\text{data}}}\|\nabla_x l(x)\|^2\), the squared gradient of its logit on the real data. In the Dirac GAN \(\nabla_xl=\psi\), so the penalty is \(\tfrac\gamma2\psi^2\), and it adds damping to the rotation. Intuitively R1 tells the discriminator "do not be steep around the real data", and at the equilibrium the discriminator then has no reason to leave \(\psi=0\). Bigger γ is not always better: γ = 1 gets within 2% in about 140 steps, while γ = 2 needs about 310 (overdamped). The gradient penalty of WGAN-GP and spectral normalization belong to the same family of "limit the discriminator's slope" methods.

噪声 \(\mathbf z\sim p_z\)（这里是标准高斯），生成器 \(G_\theta\) 把它映射为样本，诱导出分布 \(p_g\)；判别器 \(D_\phi(x)\in(0,1)\)。GAN 的目标是一个极小极大问题：

Noise \(\mathbf z\sim p_z\) (a standard Gaussian here) is mapped by the generator \(G_\theta\) to a sample, inducing a distribution \(p_g\); the discriminator is \(D_\phi(x)\in(0,1)\). The GAN objective is a minimax problem:

对 \(\phi\) 而言这就是二分类交叉熵取负（真样本标签 1，假样本标签 0，16.1 节），所以“最大化 \(V\)”就是训练一个分类器。生成器只出现在第二项里。

For \(\phi\) this is just the negative binary cross-entropy (label 1 for real, 0 for fake; Lesson 16.1), so "maximize \(V\)" means training a classifier. The generator appears only in the second term.

固定 \(G\)，把第二项改写成对 \(p_g\) 的期望：\(V=\int\big[p_{\text{data}}(x)\log D(x)+p_g(x)\log(1-D(x))\big]dx\)。假设 \(D\) 可以是任意函数，就能逐点最大化：对 \(a\log y+b\log(1-y)\) 求导，\(a/y-b/(1-y)=0\) 得 \(y=a/(a+b)\)，这就是第 2 节的 \(\hl{dstar}{D^*}\)。代回去：

Fix \(G\) and write the second term as an expectation over \(p_g\): \(V=\int\big[p_{\text{data}}(x)\log D(x)+p_g(x)\log(1-D(x))\big]dx\). If \(D\) can be any function, maximize pointwise: differentiating \(a\log y+b\log(1-y)\) gives \(a/y-b/(1-y)=0\), so \(y=a/(a+b)\), the \(\hl{dstar}{D^*}\) of Section 2. Substituting back:

（在每个对数里乘上再除以 2，得到 \(-\log2\) 两次和两个对 \(m\) 的 KL。）KL 非负且只在两分布相同时为 0（4.11 节），所以 \(\mathrm{JS}\ge0\)，等号当且仅当 \(p_g=p_{\text{data}}\)：全局最优解唯一，就是生成器完美复制数据分布，此时 \(D^*=\tfrac12\)、\(V=-\log4\)（FIG 1 的第二个任务）。JS 对称、有界；两分布支撑不相交时它恒等于 \(\log2\)，对参数的导数为零（FIG 1 的第一个任务、FIG 3）。这一切都假设判别器是“任意函数”且训练到了最优：真实的判别器是一个有限的网络，只走了几步，它近似的是一个被平滑过的 JS。

(Multiply and divide by 2 inside each log to get \(-\log2\) twice plus two KLs against \(m\).) KL is non-negative and zero only for identical distributions (Lesson 4.11), so \(\mathrm{JS}\ge0\) with equality if and only if \(p_g=p_{\text{data}}\): the global optimum is unique, and it is the generator reproducing the data distribution exactly, with \(D^*=\tfrac12\) and \(V=-\log4\) (the second task of FIG 1). JS is symmetric and bounded; when the two supports are disjoint it is constant at \(\log2\) and its derivative with respect to the parameters is zero (the first task of FIG 1, and FIG 3). All of this assumes the discriminator is "any function" trained to optimality: a real discriminator is a finite network that took only a few steps, and what it approximates is a smoothed JS.

写 \(l_\phi(x)\) 为判别器的 logit，\(D=\sigma(l)\)。生成器损失 \(L_G=\mathbb E_{\mathbf z}[\ell(l_\phi(G_\theta(\mathbf z)))]\)，由链式法则（2.2、16.3 节）：

Write \(l_\phi(x)\) for the discriminator's logit, \(D=\sigma(l)\). The generator loss is \(L_G=\mathbb E_{\mathbf z}[\ell(l_\phi(G_\theta(\mathbf z)))]\), and by the chain rule (Lessons 2.2 and 16.3):

\(\ell'(l)\) 是每个样本的一个标量权重；转置的雅可比 \((\partial G/\partial\theta)^\top\) 乘一个向量，就是穿过 \(G\) 的向量-雅可比积（VJP，16.3 节）。实现上就是：判别器做一次前向、一次反向，但只要它对输入的梯度（R101.NN 的 backward 正好返回 \(\partial L/\partial x\)），把判别器自己累积的参数梯度丢掉；再把这个向量当作生成器输出的上游梯度做一次反向传播。两种损失的区别只在标量权重：原始损失乘 \(D\)，非饱和损失乘 \(1-D\)，FIG 2 的读数同时给出两者。Arjovsky 与 Bottou（2017）进一步指出，判别器最优时非饱和损失的梯度等于 \(\nabla_\theta\big[D_{\mathrm{KL}}(p_g\|p_{\text{data}})-2\,\mathrm{JS}\big]\)：其中反向 KL 惩罚“在没有数据的地方放样本”，却几乎不惩罚“漏掉一个模式”（4.11 节的“抓峰”），这是模式坍缩的另一个来源。

Here \(\ell'(l)\) is one scalar weight per sample, and the transposed Jacobian \((\partial G/\partial\theta)^\top\) times a vector is the vector–Jacobian product (VJP) through \(G\) of Lesson 16.3. In code: run the discriminator forward and backward, but keep only its gradient with respect to the input (R101.NN's backward returns exactly \(\partial L/\partial x\)) and discard the parameter gradients it accumulated; then use that vector as the upstream gradient for one backward pass through the generator. The two losses differ only in the scalar weight: \(D\) for the original loss, \(1-D\) for the non-saturating one, and FIG 2's readout shows both. Arjovsky and Bottou (2017) further showed that with an optimal discriminator the non-saturating gradient equals \(\nabla_\theta\big[D_{\mathrm{KL}}(p_g\|p_{\text{data}})-2\,\mathrm{JS}\big]\): the reverse KL in it punishes "samples where there is no data" but hardly punishes "a missing mode" (the "mode seeking" of Lesson 4.11), another source of mode collapse.

每次迭代：(1) 判别器做 \(k\) 步：采样一批真样本 \(\{x_i\}\) 和噪声 \(\{\mathbf z_i\}\)，沿 \(\nabla_\phi\frac1n\sum_i\big[\log D(x_i)+\log(1-D(G(\mathbf z_i)))\big]\) 上升；(2) 生成器做 1 步：采样新的 \(\{\mathbf z_i\}\)，沿 \(-\nabla_\theta\frac1n\sum_i\ell(l(G(\mathbf z_i)))\) 下降。原论文的实验取 \(k=1\)，FIG 2 也是。DCGAN（Radford 等，2016）总结的经验，如 Adam 取 \(\beta_1=0.5\)（比默认的 0.9 少一些动量，因为“目标”一直在动），一直沿用至今，本页也这样设。两个网络的学习率可以不同（TTUR，Heusel 等，2017）。

Each iteration: (1) \(k\) discriminator steps: sample a batch of real \(\{x_i\}\) and noise \(\{\mathbf z_i\}\) and ascend \(\nabla_\phi\frac1n\sum_i\big[\log D(x_i)+\log(1-D(G(\mathbf z_i)))\big]\); (2) one generator step: sample fresh \(\{\mathbf z_i\}\) and descend \(-\nabla_\theta\frac1n\sum_i\ell(l(G(\mathbf z_i)))\). The original paper's experiments used \(k=1\), as does FIG 2. Rules of thumb collected by DCGAN (Radford et al., 2016), such as Adam with \(\beta_1=0.5\) (less momentum than the default 0.9, because the "target" keeps moving), are still in use; this page uses them too. The two networks may use different learning rates (TTUR; Heusel et al., 2017).

Wasserstein-1 距离（“推土机距离”）是把一堆土 \(p\) 搬成 \(q\) 的最小平均搬运距离；Kantorovich-Rubinstein 对偶把它写成对所有 1-Lipschitz 函数 \(f\) 取上确界：

The Wasserstein-1 distance ("earth mover's distance") is the smallest average distance needed to move a pile of earth \(p\) into the shape \(q\); Kantorovich-Rubinstein duality writes it as a supremum over all 1-Lipschitz functions \(f\):

WGAN 用一个网络（“评论家”，输出不过 sigmoid）近似这个 \(f\)，评论家最大化上式，生成器最小化 \(-\mathbb E_{\mathbf z}[f(G(\mathbf z))]\)。难点是 Lipschitz 约束：原文把权重裁剪到 \([-c,c]\)（粗糙，容量受限），WGAN-GP（Gulrajani 等，2017）改为惩罚 \(\lambda\,\mathbb E_{\hat x}\big(\|\nabla_{\hat x}f(\hat x)\|-1\big)^2\)，\(\hat x\) 取在真假样本的连线上；谱归一化（Miyato 等，2018）则直接把每层权重除以它的最大奇异值（1.12 节）。FIG 3 里两个等宽高斯的最优评论家是线性函数 \(f(x)=\pm x\)，所以梯度恒为 1，不论分布离得多远。注意梯度惩罚需要“对梯度再求梯度”（二阶反向传播），本页的小工具箱不支持，所以 FIG 2 只用了标准的两种损失。

The WGAN approximates this \(f\) with a network (the "critic", whose output has no sigmoid); the critic maximizes the expression above and the generator minimizes \(-\mathbb E_{\mathbf z}[f(G(\mathbf z))]\). The hard part is the Lipschitz constraint: the original paper clips the weights to \([-c,c]\) (crude, and it limits capacity); WGAN-GP (Gulrajani et al., 2017) instead penalizes \(\lambda\,\mathbb E_{\hat x}\big(\|\nabla_{\hat x}f(\hat x)\|-1\big)^2\) with \(\hat x\) on lines between real and fake samples; spectral normalization (Miyato et al., 2018) divides each layer's weights by their largest singular value (Lesson 1.12). In FIG 3 the optimal critic between two equal-width Gaussians is the linear function \(f(x)=\pm x\), so its gradient is always 1, however far apart the distributions are. Note that a gradient penalty needs "the gradient of a gradient" (second-order backpropagation), which this page's small toolkit does not support, so FIG 2 sticks to the two standard losses.

把两人的参数拼成 \(\mathbf w=(\phi,\theta)\)，同时梯度法就是 \(\mathbf w\leftarrow\mathbf w+\eta\,\mathbf v(\mathbf w)\)，\(\mathbf v=(\nabla_\phi V,\,-\nabla_\theta V)\)。在平衡点处，它的收敛性由 \(\mathbf v\) 的雅可比 \(J\) 的特征值 \(\lambda\) 决定：离散更新的放大倍数是 \(|1+\eta\lambda|\)。普通的最小化里 \(J\) 是负的 Hessian，特征值为实数且非正；零和博弈里 \(J\) 有反对称部分，特征值可以是纯虚数，于是 \(|1+\eta\lambda|=\sqrt{1+\eta^2|\lambda|^2}>1\)，无论 η 多小都发散，只是慢一些。Dirac GAN 里 \(\lambda=\pm\tfrac i2\)，同时更新每步放大 \(\sqrt{1+\eta^2/4}\)（η = 0.1 时 1.00125）；交替更新的矩阵行列式为 1（γ = 0 时），特征值落在单位圆上；R1 把 \(J\) 的实部变成 \(-\gamma/2\)，从而收敛。

Stack both players' parameters into \(\mathbf w=(\phi,\theta)\); simultaneous gradient play is \(\mathbf w\leftarrow\mathbf w+\eta\,\mathbf v(\mathbf w)\) with \(\mathbf v=(\nabla_\phi V,\,-\nabla_\theta V)\). At an equilibrium, convergence is decided by the eigenvalues \(\lambda\) of the Jacobian \(J\) of \(\mathbf v\): the discrete update amplifies by \(|1+\eta\lambda|\). In ordinary minimization \(J\) is the negative Hessian, with real non-positive eigenvalues; in a zero-sum game \(J\) has an antisymmetric part and its eigenvalues can be purely imaginary, so \(|1+\eta\lambda|=\sqrt{1+\eta^2|\lambda|^2}>1\): it diverges for every η, just more slowly for small ones. In the Dirac GAN \(\lambda=\pm\tfrac i2\), so simultaneous updates amplify by \(\sqrt{1+\eta^2/4}\) per step (1.00125 at η = 0.1); the alternating update matrix has determinant 1 (for γ = 0), with eigenvalues on the unit circle; R1 gives \(J\) a real part of \(-\gamma/2\), and it converges.

给两个网络都喂一个条件 \(c\)（类别、图像、机器人观测）：\(G(\mathbf z,c)\)、\(D(x,c)\)，目标中的期望变成对 \((x,c)\) 的联合期望（Mirza 与 Osindero，2014）。同样的推导给出 \(D^*(x,c)=p_{\text{data}}(x|c)/(p_{\text{data}}(x|c)+p_g(x|c))\)，最优解是对每个 \(c\) 都有 \(p_g(\cdot|c)=p_{\text{data}}(\cdot|c)\)：一个条件分布的采样器，而不是条件均值。pix2pix（Isola 等，2017）用它做图像到图像的翻译，下一节用它生成多模态动作 \(\hl{cond}{G(\mathbf z,c)}\)。

Feed both networks a condition \(c\) (a class, an image, a robot observation): \(G(\mathbf z,c)\) and \(D(x,c)\), and the expectations in the objective become joint expectations over \((x,c)\) (Mirza and Osindero, 2014). The same derivation gives \(D^*(x,c)=p_{\text{data}}(x|c)/(p_{\text{data}}(x|c)+p_g(x|c))\), and the optimum has \(p_g(\cdot|c)=p_{\text{data}}(\cdot|c)\) for every \(c\): a sampler of the conditional distribution, not the conditional mean. pix2pix (Isola et al., 2017) uses it for image-to-image translation, and the next section uses it to generate multimodal actions \(\hl{cond}{G(\mathbf z,c)}\).

一个移动机器人（或机械臂末端）要从下方走到上方，中间有一个障碍物，它的横向位置 \(c\) 每次不同。示教者有时从左绕、有时从右绕：障碍物偏右时多半从左绕（概率 \(\sigma(3c)\)），偏左时多半从右绕。示教给出的是经过障碍物那一行时的横向位置 \(a=c\pm0.8\)（加一点噪声）。这是一个典型的多模态条件分布 \(p(a\,|\,c)\)，和 19.1 节 FIG 6 里条件 VAE 解决的是同一类任务；这里换成对抗训练，看 GAN 能否同样学出两个分支。下图同时训练两个模型：一个用 MSE 回归 \(a\)（就是 22.1 节的行为克隆），一个条件 GAN，\(G(\mathbf z,c)\) 是 2→24→24→1 的 MLP（1 维噪声加 1 维条件），\(D(a,c)\) 是 2→24→24→1。

A mobile robot (or an arm's end effector) must travel from bottom to top past an obstacle whose lateral position \(c\) changes every time. The demonstrator sometimes passes on the left and sometimes on the right: mostly on the left when the obstacle sits to the right (probability \(\sigma(3c)\)), mostly on the right when it sits to the left. The demonstrations record the lateral position \(a=c\pm0.8\) (plus a little noise) at the obstacle's row. This is a typical multimodal conditional distribution \(p(a\,|\,c)\), the same kind of task the conditional VAE of Lesson 19.1 (FIG 6) solved; here adversarial training takes its place, to see whether a GAN learns both branches too. The figure trains two models side by side: one regresses \(a\) with MSE (the behaviour cloning of Lesson 22.1), the other is a conditional GAN, with \(G(\mathbf z,c)\) a 2→24→24→1 MLP (1D noise plus the 1D condition) and \(D(a,c)\) a 2→24→24→1 MLP.

MSE 的最优解是条件均值 \(\mathbb E[a|c]=c+0.8\,(1-2\sigma(3c))\)，在 \(|c|<0.42\) 时都落在碰撞带里，所以无论训练多久它在约 42% 的 \(c\) 上都会撞。cGAN 学的是整个条件分布：它在两个分支之间偶尔留下几个“桥”上的样本（FIG 2 里的同一现象），这就是残余的那一点碰撞率。在 \(c=0.5\) 处，数据从左绕的比例是 \(\sigma(1.5)\approx0.82\)；默认种子下 cGAN 学到的比例在 0.74 左右（读数实时给出），两个分支都有，但比例只是近似。多模态动作和它对模仿学习的意义，22.1 节和 22.2 节会再系统地比较 MSE、ACT 和 Diffusion Policy。

The MSE optimum is the conditional mean \(\mathbb E[a|c]=c+0.8\,(1-2\sigma(3c))\), which lies inside the collision band whenever \(|c|<0.42\), so however long it trains it collides on about 42% of the values of \(c\). The cGAN learns the whole conditional distribution: it occasionally leaves a few samples on a "bridge" between the two branches (the same phenomenon as in FIG 2), which is the small residual collision rate. At \(c=0.5\) the data pass on the left with probability \(\sigma(1.5)\approx0.82\); with the default seed the cGAN's learned fraction is around 0.74 (the readout shows it live): both branches are there, but the proportions are only approximate. Lessons 22.1 and 22.2 return to multimodal actions and what they mean for imitation learning, comparing MSE, ACT and Diffusion Policy systematically.