扩散与 Score-based 模型Diffusion & Score-Based Models

让人示教一个机械臂去抓桌上的杯子。同一个画面下，示教者有时抓左边那只，有时抓右边那只；绕开障碍物时，有时从上面绕，有时从下面绕。示教数据里的动作分布是多模态的。用均方误差去回归动作（16.2 节），学到的是条件均值：两只杯子的中点，或者正好穿过障碍物的那条“平均路线”。机器人需要的不是平均，而是一个能从整个动作分布里采样的模型：每次给出一个具体、可行、属于某个模式的动作。

前面几节的生成模型各有短板。VAE（19.1 节）的样本偏模糊；GAN（19.2 节）训练不稳，还会丢模式；normalizing flow（19.3 节）要求网络可逆，结构受限；自回归模型（19.4 节）要把连续动作切成 token；Energy-based 模型（19.5 节）训练和采样都要跑 MCMC。扩散模型把训练变成一个普通的回归问题，把生成变成很多步“去一点噪声”，既稳定又能覆盖所有模式，还能直接扩展到高维：几百万像素的图像，或者 16 步 × 7 维的一整段机械臂动作。

今天的图像、视频生成模型大多是扩散模型或它的近亲；在机器人里，Diffusion Policy（Chi 等，2023）用它生成动作序列，Diffuser（Janner 等，2022）用它生成整条轨迹。本节讲这台“引擎”本身：前向过程、score、训练目标、采样器、噪声日程和引导；22.2 节再讲怎样把它装进策略里。

Teach a robot arm by demonstration to pick up a cup on the table. In the same scene the demonstrator sometimes takes the left cup and sometimes the right one; to get around an obstacle, sometimes over it and sometimes under. The distribution of demonstrated actions is multimodal. Regressing the action with a squared error (Lesson 16.2) learns the conditional mean: the midpoint between the two cups, or the "average route" straight through the obstacle. What the robot needs is not the average but a model it can sample from: each time, one concrete, feasible action that belongs to one of the modes.

The generative models so far each have a weak spot. VAEs (Lesson 19.1) give blurry samples; GANs (Lesson 19.2) are unstable to train and drop modes; normalizing flows (Lesson 19.3) need invertible networks, which constrains the architecture; autoregressive models (Lesson 19.4) have to cut continuous actions into tokens; energy-based models (Lesson 19.5) need MCMC both for training and for sampling. A diffusion model turns training into an ordinary regression problem and generation into many steps of "remove a little noise". It is stable, it covers all modes, and it scales directly to high dimensions: images of millions of pixels, or a whole chunk of arm actions of 16 steps × 7 dimensions.

Most of today's image and video generators are diffusion models or close relatives; in robotics, Diffusion Policy (Chi et al., 2023) uses one to generate action sequences, and Diffuser (Janner et al., 2022) to generate whole trajectories. This lesson is about the engine itself: the forward process, the score, the training objective, the samplers, the noise schedules and guidance. Lesson 22.2 then fits it into a policy.

本节的数据是示教者让夹爪去的位置 \((x,y)\)，已经归一化到大约单位尺度（第 7 节会说为什么必须归一化）。默认的数据集是“4 个杯子”：四个位置被选中的频率分别是 40%、30%、20%、10%，每个位置周围有一点散布 \(\sigma_{\mathrm d}=0.16\)。另外两个数据集是“圆形盖沿”（沿一圈的 8 个抓取点）和“两条弧形把手”。它们都是高斯混合，所以下面每一个 score、每一个后验都能精确算出来，网络学得好不好也能和真值比。

前向过程每一步都把点往原点缩一点，再加一点高斯噪声：\(\mathbf x_t=\sqrt{1-\beta_t}\,\mathbf x_{t-1}+\sqrt{\beta_t}\,\boldsymbol\varepsilon\)。缩一点是为了让总方差保持在 1 附近，不让噪声越加越大。高斯加高斯还是高斯（4.4 节），所以走了 \(t\) 步之后有一个闭式：\(\mathbf x_t=\sqrt{\bar\alpha_t}\,\mathbf x_0+\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon\)，其中 \(\bar\alpha_t\) 从 1 单调降到几乎 0。不需要真的一步步模拟，任何 \(t\) 都能一步跳到。拖动下面的 \(t\)，看四个杯子怎样被“磨”成一团标准高斯。

时间方向的约定：本节沿用 DDPM 的写法，\(t\) 数的是加噪的步数：\(t=0\) 是数据 \(\mathbf x_0\)，\(t=T=100\) 几乎是纯噪声，生成时 \(t\) 从 \(T\) 倒着走回 0。19.7 节的 flow matching 用的是相反的约定（flow matching 文献和 π0 都这样写）：\(t\in[0,1]\)，\(t=0\) 是噪声，\(t=1\) 是数据，那里的 \(\mathbf x_0\) 是噪声（本节的 \(\boldsymbol\varepsilon\)，约等于 \(\mathbf x_T\)），\(\mathbf x_1\) 才是数据（本节的 \(\mathbf x_0\)）；两边的时间大致对应为 \(t_{19.7}\approx1-t/T\)。

The data in this lesson are the positions \((x,y)\) a demonstrator sent the gripper to, already normalized to roughly unit scale (Section 7 says why this is necessary). The default dataset is "4 cups": the four positions are chosen 40%, 30%, 20% and 10% of the time, each with a little spread \(\sigma_{\mathrm d}=0.16\). The other two are a "round rim" (8 grasp points around a circle) and "two arc handles". All of them are Gaussian mixtures, so every score and every posterior below can be computed exactly, and what the network learns can be compared with the truth.

Each step of the forward process shrinks the point a little toward the origin and adds a little Gaussian noise: \(\mathbf x_t=\sqrt{1-\beta_t}\,\mathbf x_{t-1}+\sqrt{\beta_t}\,\boldsymbol\varepsilon\). The shrinking keeps the total variance near 1, so the noise does not keep growing. Gaussian plus Gaussian is Gaussian (Lesson 4.4), so after \(t\) steps there is a closed form, \(\mathbf x_t=\sqrt{\bar\alpha_t}\,\mathbf x_0+\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon\), with \(\bar\alpha_t\) falling monotonically from 1 to almost 0. Nothing has to be simulated step by step; any \(t\) can be reached in one jump. Drag \(t\) below and watch the four cups get ground into one standard Gaussian blob.

Time convention: this lesson follows DDPM, where \(t\) counts noising steps: \(t=0\) is the data \(\mathbf x_0\), \(t=T=100\) is almost pure noise, and generation runs \(t\) backwards from \(T\) to 0. The flow matching of Lesson 19.7 uses the opposite convention (as the flow-matching literature and π0 do): \(t\in[0,1]\) with \(t=0\) the noise and \(t=1\) the data, so there \(\mathbf x_0\) is the noise (this lesson's \(\boldsymbol\varepsilon\), roughly \(\mathbf x_T\)) and \(\mathbf x_1\) is the data (this lesson's \(\mathbf x_0\)); the two time axes correspond roughly as \(t_{19.7}\approx1-t/T\).

要倒着走，就得回答：看到一个带噪声的点 \(\mathbf x_t\)，它原来最可能在哪？按平方误差算，最好的猜测是后验均值 \(\hat{\mathbf x}_0=\mathbb E[\mathbf x_0\mid\mathbf x_t]\)（最小均方误差估计，即 MMSE；后验见 4.6 节）。18.2 节已经给出了 Tweedie 公式：对高斯破坏，后验均值等于“输入加上噪声方差乘 score”。换成这里的缩放记号是 \(\sqrt{\bar\alpha_t}\,\hat{\mathbf x}_0=\mathbf x_t+(1-\bar\alpha_t)\nabla\log p_t(\mathbf x_t)\)。所以“学会去噪”和“学会 score \(\nabla\log p_t\)”是同一件事，score 就是从 \(\mathbf x_t\) 指向（缩放后的）最佳去噪猜测的方向。

下图把这件事画出来。注意后验 \(p(\mathbf x_0\mid\mathbf x_t)\) 本身是多模态的：噪声大时，一个点可能来自好几个杯子，后验均值就落在杯子之间的空地上，那里根本没有数据。这就是为什么扩散模型不能一步生成：一步去噪只能给出平均，必须走很多小步，每一步只需要局部的方向，后验在这一路上逐渐“下决心”。

To walk backwards we must answer: given a noisy point \(\mathbf x_t\), where did it most likely come from? Under squared error the best guess is the posterior mean \(\hat{\mathbf x}_0=\mathbb E[\mathbf x_0\mid\mathbf x_t]\) (the minimum-mean-squared-error, or MMSE, estimate; posteriors are those of Lesson 4.6). Lesson 18.2 already gave Tweedie's formula: for Gaussian corruption the posterior mean is "the input plus the noise variance times the score". In the scaled notation here it reads \(\sqrt{\bar\alpha_t}\,\hat{\mathbf x}_0=\mathbf x_t+(1-\bar\alpha_t)\nabla\log p_t(\mathbf x_t)\). So "learning to denoise" and "learning the score \(\nabla\log p_t\)" are the same thing, and the score is the direction from \(\mathbf x_t\) toward the (rescaled) best denoising guess.

The figure below draws exactly this. Notice that the posterior \(p(\mathbf x_0\mid\mathbf x_t)\) is itself multimodal: when the noise is large, one point could have come from several cups, and the posterior mean lands in the empty space between them, where there is no data at all. This is why a diffusion model cannot generate in one step: one denoising step can only give the average. It has to take many small steps, each needing only the local direction, and along the way the posterior gradually "makes up its mind".

现在不再用精确的 score，而是让网络去学。DDPM（Ho 等，2020）的训练循环只有四行：从数据里取一个 \(\mathbf x_0\)，均匀地抽一个噪声步 \(t\in\{1,\dots,T\}\)，抽一份噪声 \(\boldsymbol\varepsilon\sim\mathcal N(0,I)\)，按闭式算出 \(\mathbf x_t\)，然后让网络 \(\boldsymbol\varepsilon_\theta(\mathbf x_t,t)\) 回归 \(\boldsymbol\varepsilon\)，损失是均方误差。这里 \(T=100\)，网络是一个 9 → 64 → 64 → 2 的 ReLU MLP（4930 个参数）：输入是 \(\mathbf x_t\) 的两个坐标加上 7 个时间特征（\(\pi t/T\)、\(2\pi t/T\)、\(4\pi t/T\) 的正弦和余弦，以及一个线性斜坡），输出是预测的噪声。优化器是 Adam，学习率 \(2\times10^{-3}\)，批量 64（6.3 节）。

生成时用第 4 节要讲的 DDPM 采样器，在 100 个训练噪声水平里等间隔取 50 个，从 300 个纯噪声点出发往回走。为了让你看到训练过程，每一批样本都从当时权重的一份快照开始采样，采完立刻开始下一批。还有两处细节：EMA（指数滑动平均）权重 \(\bar\theta\leftarrow\bar\theta+0.01(\theta-\bar\theta)\)，几乎所有扩散模型都用它来采样；损失下限：即使用精确的最优去噪器，损失也不会是 0（第 5 节解释），图里绿色虚线就是精确 \(\boldsymbol\varepsilon^\star\) 在同一个测试集上的损失。

Now we stop using the exact score and let a network learn it. The training loop of DDPM (Ho et al., 2020) is four lines: take an \(\mathbf x_0\) from the data, draw a noise step \(t\in\{1,\dots,T\}\) uniformly, draw noise \(\boldsymbol\varepsilon\sim\mathcal N(0,I)\), compute \(\mathbf x_t\) from the closed form, and regress the network \(\boldsymbol\varepsilon_\theta(\mathbf x_t,t)\) onto \(\boldsymbol\varepsilon\) with a squared-error loss. Here \(T=100\) and the network is a 9 → 64 → 64 → 2 ReLU MLP (4930 parameters): the input is the two coordinates of \(\mathbf x_t\) plus 7 time features (sine and cosine of \(\pi t/T\), \(2\pi t/T\) and \(4\pi t/T\), and a linear ramp), the output is the predicted noise. The optimizer is Adam with learning rate \(2\times10^{-3}\) and batch size 64 (Lesson 6.3).

To generate, we use the DDPM sampler of Section 4, taking 50 of the 100 training noise levels at even spacing and walking back from 300 points of pure noise. So that you can watch training happen, each batch of samples starts from a snapshot of the weights at that moment, and the next batch starts as soon as one finishes. Two more details: EMA (exponential moving average) weights \(\bar\theta\leftarrow\bar\theta+0.01(\theta-\bar\theta)\), which nearly every diffusion model samples with; and the loss floor: even the exact optimal denoiser does not reach zero loss (Section 5 explains why), and the green dashed line is the loss of the exact \(\boldsymbol\varepsilon^\star\) on the same test set.

有了 \(\boldsymbol\varepsilon_\theta\)，怎样从 \(\mathbf x_T\sim\mathcal N(0,I)\) 走回 \(\mathbf x_0\)？每一步都先用网络猜出 \(\hat{\mathbf x}_0=(\mathbf x_t-\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon_\theta)/\sqrt{\bar\alpha_t}\)，然后决定下一个点 \(\mathbf x_s\)（\(s<t\)）放在哪里。DDPM（祖先采样）把 \(\mathbf x_s\) 从“已知 \(\mathbf x_t\) 和 \(\hat{\mathbf x}_0\) 时 \(\mathbf x_s\) 的高斯后验”里抽出来，每步都加新的噪声；DDIM（Song 等，2021）取同一族里噪声为零的那个成员：\(\mathbf x_s=\sqrt{\bar\alpha_s}\,\hat{\mathbf x}_0+\sqrt{1-\bar\alpha_s}\,\boldsymbol\varepsilon_\theta\)，是确定性的。两者用同一个训练好的网络，都可以跳着走：只访问 \(T\) 个训练噪声水平中的 \(N\) 个。网络调用次数就是 \(N\)，这是推理耗时的主要来源。

为了把“采样器”和“网络学得好不好”分开，下图用精确的 \(\boldsymbol\varepsilon^\star\) 代替网络：所有误差都来自采样器本身的离散化。

Given \(\boldsymbol\varepsilon_\theta\), how do we walk from \(\mathbf x_T\sim\mathcal N(0,I)\) back to \(\mathbf x_0\)? At each step the network first guesses \(\hat{\mathbf x}_0=(\mathbf x_t-\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon_\theta)/\sqrt{\bar\alpha_t}\), and then we decide where to put the next point \(\mathbf x_s\) (\(s<t\)). DDPM (ancestral sampling) draws \(\mathbf x_s\) from the Gaussian posterior of \(\mathbf x_s\) given \(\mathbf x_t\) and \(\hat{\mathbf x}_0\), adding fresh noise at every step; DDIM (Song et al., 2021) takes the zero-noise member of the same family, \(\mathbf x_s=\sqrt{\bar\alpha_s}\,\hat{\mathbf x}_0+\sqrt{1-\bar\alpha_s}\,\boldsymbol\varepsilon_\theta\), which is deterministic. Both use the same trained network, and both can skip: they visit only \(N\) of the \(T\) training noise levels. The number of network calls is \(N\), and that is where most of the inference time goes.

To separate "the sampler" from "how well the network learned", the figure below uses the exact \(\boldsymbol\varepsilon^\star\) in place of a network: every error comes from the sampler's own discretization.

几个数字（600 个粒子，默认噪声）：“4 个杯子”上，余弦日程的 DDPM 在 10 步时还有 98% 的样本落在数据上，5 步 92%，2 步只剩 34%；DDIM 在 5 步时是 93%，10 步 99%。在这个二维的小例子里两者差别不大；在图像上，DDIM 的少步数优势更明显。日程的影响在“圆形盖沿”上最清楚：DDPM 5 步，余弦日程 78%，线性日程只有 32%；10 步时是 94% 对 80%。线性日程把大量步数花在 \(\bar\alpha_t\approx0\) 的纯噪声区（FIG 1），步数一少，真正决定形状的中低噪声段就分不到几步了。

Some numbers (600 particles, the default noise): on "4 cups" with the cosine schedule, DDPM still puts 98% of its samples on the data at 10 steps, 92% at 5 steps and only 34% at 2 steps; DDIM gives 93% at 5 steps and 99% at 10. In this small 2D example the two are close; on images DDIM's advantage at few steps is larger. The schedule matters most on the "round rim": DDPM with 5 steps reaches 78% with the cosine schedule but only 32% with the linear one; at 10 steps it is 94% against 80%. The linear schedule spends many of its steps in the pure-noise region where \(\bar\alpha_t\approx0\) (FIG 1), so with few steps the low and medium noise levels, where the shape is actually decided, get hardly any.

机器人要的通常是条件分布：给定指令 \(c\)（“拿马克杯”还是“拿碗”）或者给定观测，生成动作。最直接的做法是把 \(c\) 也输进去，训练 \(\boldsymbol\varepsilon_\theta(\mathbf x_t,t,c)\)。实际中这样学到的条件模型常常“不够听话”：一部分样本无视条件。Classifier-free guidance（Ho 与 Salimans，2021）的办法是：训练时以一定概率（常见 10% 到 20%）把 \(c\) 换成“空条件”\(\varnothing\)，于是同一个网络既是条件模型也是无条件模型；采样时把两者外推：

A robot usually needs a conditional distribution: generate an action given a command \(c\) ("pick the mug" or "pick the bowl") or given an observation. The direct way is to feed \(c\) in as well and train \(\boldsymbol\varepsilon_\theta(\mathbf x_t,t,c)\). In practice a conditional model learned this way is often "not obedient enough": part of its samples ignore the condition. Classifier-free guidance (Ho and Salimans, 2021) fixes this as follows: during training, replace \(c\) by an "empty condition" \(\varnothing\) with some probability (10% to 20% is common), so that one network is both the conditional and the unconditional model; at sampling time, extrapolate from one to the other:

\(w=0\) 是无条件模型，\(w=1\) 是条件模型本身，\(w>1\) 是“比条件模型更条件”。下图沿用“4 个杯子”的数据，但把四个物体分成两类：1 号和 4 号是马克杯（紫），2 号和 3 号是碗（绿）。条件模型是精确的，但故意做成“漏标”的：它以权重 \(\lambda\) 忽略指令，即 \(p_\theta(\mathbf x\mid c)=(1-\lambda)\,p(\mathbf x\mid c)+\lambda\,p(\mathbf x)\)，模拟一个没学好条件的网络。

\(w=0\) is the unconditional model, \(w=1\) the conditional model itself, and \(w>1\) is "more conditional than the conditional model". The figure below reuses the "4 cups" data but splits the four objects into two classes: objects 1 and 4 are mugs (purple), 2 and 3 are bowls (green). The conditional model is exact but deliberately "leaky": with weight \(\lambda\) it ignores the command, \(p_\theta(\mathbf x\mid c)=(1-\lambda)\,p(\mathbf x\mid c)+\lambda\,p(\mathbf x)\), imitating a network that has not quite learned the condition.

给定方差日程 \(\beta_1,\dots,\beta_T\in(0,1)\)，令 \(\alpha_t=1-\beta_t\)、\(\bar\alpha_t=\prod_{s=1}^t\alpha_s\)。前向过程是一条马尔可夫链：

Given a variance schedule \(\beta_1,\dots,\beta_T\in(0,1)\), let \(\alpha_t=1-\beta_t\) and \(\bar\alpha_t=\prod_{s=1}^t\alpha_s\). The forward process is a Markov chain:

用归纳法：若 \(\mathbf x_{t-1}=\sqrt{\bar\alpha_{t-1}}\mathbf x_0+\sqrt{1-\bar\alpha_{t-1}}\,\boldsymbol\varepsilon'\)，则 \(\mathbf x_t=\sqrt{\alpha_t\bar\alpha_{t-1}}\,\mathbf x_0+\sqrt{\alpha_t(1-\bar\alpha_{t-1})}\,\boldsymbol\varepsilon'+\sqrt{\beta_t}\,\boldsymbol\varepsilon''\)。两份独立高斯噪声之和仍是高斯（4.4 节），方差相加：\(\alpha_t-\alpha_t\bar\alpha_{t-1}+1-\alpha_t=1-\bar\alpha_t\)。所以 \(\mathbf x_t=\sqrt{\bar\alpha_t}\,\mathbf x_0+\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon\)，\(\boldsymbol\varepsilon\sim\mathcal N(0,I)\)。如果 \(\mathbf x_0\) 的协方差是 \(I\)，那么每个 \(\mathbf x_t\) 的协方差也是 \(I\)，这就是“方差保持”（variance preserving）的含义。\(\bar\alpha_T\approx0\) 时 \(q(\mathbf x_T\mid\mathbf x_0)\approx\mathcal N(0,I)\)，与 \(\mathbf x_0\) 无关，所以生成可以从标准高斯开始。本页的余弦日程有 \(\bar\alpha_T\approx2.4\times10^{-7}\)，线性日程 \(\approx2.0\times10^{-5}\)。

By induction: if \(\mathbf x_{t-1}=\sqrt{\bar\alpha_{t-1}}\mathbf x_0+\sqrt{1-\bar\alpha_{t-1}}\,\boldsymbol\varepsilon'\), then \(\mathbf x_t=\sqrt{\alpha_t\bar\alpha_{t-1}}\,\mathbf x_0+\sqrt{\alpha_t(1-\bar\alpha_{t-1})}\,\boldsymbol\varepsilon'+\sqrt{\beta_t}\,\boldsymbol\varepsilon''\). A sum of two independent Gaussian noises is Gaussian (Lesson 4.4) and the variances add: \(\alpha_t-\alpha_t\bar\alpha_{t-1}+1-\alpha_t=1-\bar\alpha_t\). Hence \(\mathbf x_t=\sqrt{\bar\alpha_t}\,\mathbf x_0+\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon\) with \(\boldsymbol\varepsilon\sim\mathcal N(0,I)\). If \(\mathbf x_0\) has covariance \(I\), so does every \(\mathbf x_t\); that is what "variance preserving" means. When \(\bar\alpha_T\approx0\), \(q(\mathbf x_T\mid\mathbf x_0)\approx\mathcal N(0,I)\) regardless of \(\mathbf x_0\), so generation can start from a standard Gaussian. On this page the cosine schedule has \(\bar\alpha_T\approx2.4\times10^{-7}\) and the linear one \(\approx2.0\times10^{-5}\).

真正的反向条件 \(q(\mathbf x_{t-1}\mid\mathbf x_t)\) 依赖整个数据分布，算不出来；但多给一个 \(\mathbf x_0\)，用贝叶斯公式和高斯的乘积（4.4 节）就得到闭式：

The true reverse conditional \(q(\mathbf x_{t-1}\mid\mathbf x_t)\) depends on the whole data distribution and cannot be computed; but given \(\mathbf x_0\) as well, Bayes' rule and the product of Gaussians (Lesson 4.4) give a closed form:

模型取同样的形式 \(p_\theta(\mathbf x_{t-1}\mid\mathbf x_t)=\mathcal N(\boldsymbol\mu_\theta(\mathbf x_t,t),\sigma_t^2I)\)。把 \(\mathbf x_0=(\mathbf x_t-\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon)/\sqrt{\bar\alpha_t}\) 代进去，\(\tilde{\boldsymbol\mu}_t=\frac{1}{\sqrt{\alpha_t}}\big(\mathbf x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\boldsymbol\varepsilon\big)\)；于是让网络预测 \(\boldsymbol\varepsilon\)，再用同一个式子算 \(\boldsymbol\mu_\theta\)。和 19.1 节一样写出负对数似然的变分上界（ELBO 取负），它分解成每一步的 KL 散度（第一行）。两个高斯之间的 KL 只与均值差的平方有关（方差固定时），代入上面的参数化就是第二行（\(C_t\) 与 \(\theta\) 无关）：

The model takes the same form, \(p_\theta(\mathbf x_{t-1}\mid\mathbf x_t)=\mathcal N(\boldsymbol\mu_\theta(\mathbf x_t,t),\sigma_t^2I)\). Substituting \(\mathbf x_0=(\mathbf x_t-\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon)/\sqrt{\bar\alpha_t}\) gives \(\tilde{\boldsymbol\mu}_t=\frac{1}{\sqrt{\alpha_t}}\big(\mathbf x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\boldsymbol\varepsilon\big)\); so we let the network predict \(\boldsymbol\varepsilon\) and compute \(\boldsymbol\mu_\theta\) from the same expression. Writing the variational upper bound on the negative log-likelihood as in Lesson 19.1 (the negative ELBO), it splits into one KL divergence per step (first line). With the variances fixed, the KL between two Gaussians depends on \(\theta\) only through the squared difference of their means, and substituting the parametrization above gives the second line (\(C_t\) does not depend on \(\theta\)):

（\(C\) 里包括与 \(\theta\) 无关的 \(\mathbf x_T\) 项；\(t=1\) 的重建项另算。）Ho 等发现把每一步的权重全扔掉、只用第三行的 \(L_{\boldsymbol\varepsilon}\)（他们称为 “simple” 损失），样本反而更好：它相对地加重了噪声大的那些步。FIG 3 训练的就是 \(L_{\boldsymbol\varepsilon}\)（除以维数 2）。

(\(C\) includes the \(\mathbf x_T\) term, which does not depend on \(\theta\); the \(t=1\) reconstruction term is handled separately.) Ho et al. found that throwing all the per-step weights away and using the \(L_{\boldsymbol\varepsilon}\) of the third line (their "simple" loss) gives better samples: it puts relatively more weight on the noisier steps. FIG 3 trains exactly \(L_{\boldsymbol\varepsilon}\) (divided by the dimension, 2).

\(q(\mathbf x_t\mid\mathbf x_0)\) 是高斯，它的 score 是 \(\nabla_{\mathbf x_t}\log q(\mathbf x_t\mid\mathbf x_0)=-(\mathbf x_t-\sqrt{\bar\alpha_t}\mathbf x_0)/(1-\bar\alpha_t)=-\boldsymbol\varepsilon/\sqrt{1-\bar\alpha_t}\)。Vincent（2011）证明：对 \(\mathbf x_0\) 和噪声取期望，回归这个“条件 score”与回归边缘密度的 score \(\nabla\log p_t\) 只差一个与网络无关的常数（去噪 score matching；直接的 score matching 是 Hyvärinen 2005 的想法）。所以预测噪声就是预测 score，只差一个尺度：

\(q(\mathbf x_t\mid\mathbf x_0)\) is Gaussian, and its score is \(\nabla_{\mathbf x_t}\log q(\mathbf x_t\mid\mathbf x_0)=-(\mathbf x_t-\sqrt{\bar\alpha_t}\mathbf x_0)/(1-\bar\alpha_t)=-\boldsymbol\varepsilon/\sqrt{1-\bar\alpha_t}\). Vincent (2011) showed that, in expectation over \(\mathbf x_0\) and the noise, regressing onto this "conditional score" differs from regressing onto the score of the marginal density \(\nabla\log p_t\) only by a constant independent of the network (denoising score matching; plain score matching is Hyvärinen's 2005 idea). Predicting the noise is therefore predicting the score, up to a scale:

第一式：平方误差的最优解是条件均值；第二式是 Tweedie 公式，取 \(\mathbf x_t=\sqrt{\bar\alpha_t}\mathbf x_0+\sqrt{1-\bar\alpha_t}\boldsymbol\varepsilon\) 的条件期望即可从第一式得到。对高斯混合，后验 \(\hl{post}{p(\mathbf x_0\mid\mathbf x_t)}\) 是另一个高斯混合，每个分量的权重 \(r_k(\mathbf x_t)\) 就是 FIG 2 读数里的“后验 p(杯子 k | x_t)”，所以页面里的一切都是精确算出来的。

最优网络的损失是 \(\hl{floor}{L^\star=\mathbb E_t\,\mathbb E_{\mathbf x_t}\operatorname{tr}\operatorname{Cov}[\boldsymbol\varepsilon\mid\mathbf x_t]}\)：只要从 \(\mathbf x_t\) 推不出唯一的 \(\boldsymbol\varepsilon\)，它就大于 0。\(t\) 很小时，\(\mathbf x_t\approx\mathbf x_0\)，噪声几乎完全不可辨识，那一步的损失接近 1（每个坐标）；\(t\) 很大时，\(\mathbf x_t\approx\boldsymbol\varepsilon\)，损失接近 0。FIG 3 的“4 个杯子”上平均起来约 0.295。所以：损失曲线能告诉你网络离最优还有多远，却不能直接和别的数据集、别的日程比较；它也不衡量样本质量。

The first identity holds because the minimizer of a squared error is the conditional mean; the second is Tweedie's formula, obtained from the first by taking the conditional expectation of \(\mathbf x_t=\sqrt{\bar\alpha_t}\mathbf x_0+\sqrt{1-\bar\alpha_t}\boldsymbol\varepsilon\). For a Gaussian mixture the posterior \(\hl{post}{p(\mathbf x_0\mid\mathbf x_t)}\) is another Gaussian mixture whose component weights \(r_k(\mathbf x_t)\) are the "posterior p(cup k | x_t)" of the FIG 2 readout, which is why everything on this page can be computed exactly.

The loss of the optimal network is \(\hl{floor}{L^\star=\mathbb E_t\,\mathbb E_{\mathbf x_t}\operatorname{tr}\operatorname{Cov}[\boldsymbol\varepsilon\mid\mathbf x_t]}\): positive as long as \(\mathbf x_t\) does not determine \(\boldsymbol\varepsilon\) uniquely. At small \(t\), \(\mathbf x_t\approx\mathbf x_0\) and the noise is almost unidentifiable, so the loss of that step is close to 1 (per coordinate); at large \(t\), \(\mathbf x_t\approx\boldsymbol\varepsilon\) and the loss is close to 0. Averaged over \(t\) it is about 0.295 for "4 cups" in FIG 3. So a loss curve tells you how far the network is from optimal, but it cannot be compared directly across datasets or schedules, and it does not measure sample quality.

采样器访问时刻 \(T=\tau_N>\dots>\tau_0=0\)。从 \(t=\tau_i\) 到 \(s=\tau_{i-1}\)：先算 \(\hat{\mathbf x}_0=(\mathbf x_t-\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon_\theta)/\sqrt{\bar\alpha_t}\)，再

The sampler visits \(T=\tau_N>\dots>\tau_0=0\). From \(t=\tau_i\) to \(s=\tau_{i-1}\): first compute \(\hat{\mathbf x}_0=(\mathbf x_t-\sqrt{1-\bar\alpha_t}\,\boldsymbol\varepsilon_\theta)/\sqrt{\bar\alpha_t}\), then

DDPM 一行就是上面的后验 \(q(\mathbf x_s\mid\mathbf x_t,\hat{\mathbf x}_0)\)，相邻一步（\(s=t-1\)）时它退化为 Ho 等的原始更新 \(\mathbf x_{t-1}=\frac{1}{\sqrt{\alpha_t}}\big(\mathbf x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\boldsymbol\varepsilon_\theta\big)+\sigma_t\mathbf z\)，\(\sigma_t^2=\tilde\beta_t\)。DDIM 那一行把“当前猜的数据”和“当前猜的噪声”按 \(s\) 时刻的比例重新组合，不加新噪声。Song 等证明，DDIM 和 DDPM 对应的是同一个训练目标（所以可以共用一个网络），区别只在反向过程里注入多少新噪声。注意 \(N=1\) 时两者都输出 \(\hat{\mathbf x}_0(\mathbf x_T)\approx\mathbb E[\mathbf x_0]\)：这就是 FIG 4 里“所有样本落在数据均值上”。

The DDPM line is the posterior \(q(\mathbf x_s\mid\mathbf x_t,\hat{\mathbf x}_0)\) from above; for adjacent steps (\(s=t-1\)) it reduces to the original update of Ho et al., \(\mathbf x_{t-1}=\frac{1}{\sqrt{\alpha_t}}\big(\mathbf x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\boldsymbol\varepsilon_\theta\big)+\sigma_t\mathbf z\) with \(\sigma_t^2=\tilde\beta_t\). The DDIM line recombines "the current guess of the data" and "the current guess of the noise" in the proportions of time \(s\), adding no fresh noise. Song et al. showed that DDIM and DDPM correspond to the same training objective (so they can share one network) and differ only in how much fresh noise the reverse process injects. Note that at \(N=1\) both output \(\hat{\mathbf x}_0(\mathbf x_T)\approx\mathbb E[\mathbf x_0]\): that is "every sample lands on the data mean" in FIG 4.

让步数趋于无穷、每步的 \(\beta\) 趋于 0（\(\beta_t\approx\beta(t)\,\mathrm dt\)），前向过程变成一个随机微分方程（Song 等，ICLR 2021，“方差保持” SDE）。Anderson（1982）给出了它的时间反演，Song 等还指出存在一个与它每个时刻边缘分布都相同的确定性 ODE。下面三行依次是前向 SDE、反向 SDE 和这个“概率流 ODE”：

Let the number of steps go to infinity with each \(\beta\) going to 0 (\(\beta_t\approx\beta(t)\,\mathrm dt\)) and the forward process becomes a stochastic differential equation (Song et al., ICLR 2021, the "variance-preserving" SDE). Anderson (1982) gave its time reversal, and Song et al. pointed out a deterministic ODE with the same marginal at every time. Below, the three lines are the forward SDE, the reverse SDE and this "probability-flow ODE":

反向 SDE 里时间倒着走（从 \(t=T\) 到 0），\(\bar{\mathbf w}\) 是反向的布朗运动。DDPM 是反向 SDE 的一种离散化，DDIM 是概率流 ODE 的一种离散化（所以它是确定性的，也是可逆的：从数据可以积分回到对应的 \(\mathbf x_T\)）。把 ODE 看成一个随时间变化的向量场（3.6 节），用更好的积分器（2.5 节）就能用更少的步数采样；直接去学这个向量场，就是 19.7 节的 flow matching（它把时间反过来，从噪声 \(t=0\) 积分到数据 \(t=1\)，见第 2 节的约定）。另一条历史线索来自 Langevin 动力学：\(\mathbf x\leftarrow\mathbf x+\eta\nabla\log p(\mathbf x)+\sqrt{2\eta}\,\mathbf z\) 在步长足够小、步数足够多时从 \(p\) 采样（19.5 节）。但在数据稀疏的地方 score 学不准，Langevin 也走不动；Song 与 Ermon（2019）的 NCSN 用一串从大到小的噪声水平做“退火”Langevin：大噪声下的 score 在全空间都有定义，把样本先带到数据附近，小噪声再雕出细节。DDPM 和 NCSN 后来被统一为同一个 SDE 框架的两个特例。

In the reverse SDE time runs backwards (from \(t=T\) to 0) and \(\bar{\mathbf w}\) is a reverse-time Brownian motion. DDPM is one discretization of the reverse SDE and DDIM one discretization of the probability-flow ODE (which is why it is deterministic, and invertible: one can integrate from the data back to its \(\mathbf x_T\)). Viewing the ODE as a time-varying vector field (Lesson 3.6), better integrators (Lesson 2.5) sample in fewer steps; learning that vector field directly is the flow matching of Lesson 19.7 (which reverses time and integrates from the noise at \(t=0\) to the data at \(t=1\); see the convention note in Section 2). Another historical thread comes from Langevin dynamics: \(\mathbf x\leftarrow\mathbf x+\eta\nabla\log p(\mathbf x)+\sqrt{2\eta}\,\mathbf z\) samples from \(p\) when the step is small enough and the steps many enough (Lesson 19.5). Where data are sparse, however, the score is learned poorly and Langevin gets stuck; the NCSN of Song and Ermon (2019) runs "annealed" Langevin through a sequence of noise levels from large to small: the score at large noise is defined over the whole space and carries the samples near the data, and the small noise levels then carve the details. DDPM and NCSN were later unified as two special cases of the same SDE framework.

日程最好用信噪比 \(\mathrm{SNR}(t)=\bar\alpha_t/(1-\bar\alpha_t)\) 来理解：它从无穷大单调降到几乎 0，日程决定的是在每个 SNR 区间里花多少步。Ho 等的线性日程是 \(T=1000\) 时 \(\beta\) 从 \(10^{-4}\) 线性增到 0.02；本页 \(T=100\)，按 Nichol 与 Dhariwal 的做法把端点乘以 \(1000/T\)，即从 0.001 到 0.2。他们提出的余弦日程是 \(\bar\alpha_t=f(t)/f(0)\)，\(f(t)=\cos^2\!\big(\frac{t/T+s}{1+s}\cdot\frac\pi2\big)\)，\(s=0.008\)，\(\beta_t\) 截断在 0.999。FIG 1 显示了区别：线性日程在 \(t=27\) 时 \(\bar\alpha_t\) 已低于 0.5，余弦日程要到 \(t=50\)。数据的尺度也是日程的一部分：数据的标准差如果是 10 而不是 1，同一个 \(\bar\alpha_t\) 下的有效信噪比就大了 100 倍，整条日程都偏了。

A schedule is best understood through the signal-to-noise ratio \(\mathrm{SNR}(t)=\bar\alpha_t/(1-\bar\alpha_t)\), which falls monotonically from infinity to almost 0; the schedule decides how many steps are spent in each SNR range. The linear schedule of Ho et al. raises \(\beta\) linearly from \(10^{-4}\) to 0.02 over \(T=1000\) steps; with \(T=100\) on this page the endpoints are multiplied by \(1000/T\), as Nichol and Dhariwal do, giving 0.001 to 0.2. Their cosine schedule is \(\bar\alpha_t=f(t)/f(0)\) with \(f(t)=\cos^2\!\big(\frac{t/T+s}{1+s}\cdot\frac\pi2\big)\), \(s=0.008\), and \(\beta_t\) clipped at 0.999. FIG 1 shows the difference: under the linear schedule \(\bar\alpha_t\) is already below 0.5 at \(t=27\), under the cosine one only at \(t=50\). The scale of the data is part of the schedule too: if the data had standard deviation 10 instead of 1, the effective SNR at the same \(\bar\alpha_t\) would be 100 times larger, and the whole schedule would be off.

由贝叶斯公式 \(p_t(\mathbf x\mid c)\propto p_t(\mathbf x)\,p_t(c\mid\mathbf x)\)，对 \(\mathbf x\) 求梯度，归一化常数消失：

From Bayes' rule \(p_t(\mathbf x\mid c)\propto p_t(\mathbf x)\,p_t(c\mid\mathbf x)\); taking the gradient in \(\mathbf x\), the normalizing constant disappears:

Classifier guidance（Dhariwal 与 Nichol，2021）单独训练一个在带噪输入上工作的分类器 \(p_\phi(c\mid\mathbf x_t)\)，把它的梯度乘一个系数加进 score。Classifier-free guidance 不要分类器：按上式，两个 score 之差就是一个“隐式分类器”的梯度，\(w>1\) 等于给它额外的权重 \(w-1\)，样本被推向这个分类器最有把握的地方。直观上它像是在 \(p(\mathbf x)\,p(c\mid\mathbf x)^w\) 里采样，但这只是启发式的说法：引导后的 \(\tilde{\mathbf s}\) 一般并不是任何一族加噪分布的精确 score。（Ho 与 Salimans 的原文把系数写成 \((1+w')\varepsilon_c-w'\varepsilon_\varnothing\)，即 \(w'=w-1\)；本页沿用多数代码库里“\(w=1\) 即不引导”的写法。）FIG 5 里较少的 4 号马克杯被“挤掉”，是因为它本身少见，旁边又是更常见的碗 2，带噪时它附近的 \(p(c\mid\mathbf x)\)（\(c\) 为“马克杯”）就小。

Classifier guidance (Dhariwal and Nichol, 2021) trains a separate classifier \(p_\phi(c\mid\mathbf x_t)\) on noisy inputs and adds its gradient, times a coefficient, to the score. Classifier-free guidance needs no classifier: by the equation above, the difference of the two scores is the gradient of an "implicit classifier", and \(w>1\) gives it extra weight \(w-1\), pushing samples to where that classifier is most certain. Intuitively this is like sampling from \(p(\mathbf x)\,p(c\mid\mathbf x)^w\), but only heuristically: the guided \(\tilde{\mathbf s}\) is in general not the exact score of any family of noised distributions. (Ho and Salimans write the combination as \((1+w')\varepsilon_c-w'\varepsilon_\varnothing\), i.e. \(w'=w-1\); this page follows the convention of most codebases, where \(w=1\) means no guidance.) The rarer mug no. 4 in FIG 5 gets squeezed out because it is rare itself and sits next to the more frequent bowl no. 2, so under noise \(p(c\mid\mathbf x)\) with \(c\) = mug is small around it.

前面的“数据”是二维的点。对机器人，被生成的对象换成一整段动作：比如未来 10 个路点的位置，这是一个 20 维的向量。扩散模型并不在乎维数，前向过程、训练目标和采样器一个字都不用改；它对整段动作一起去噪，所以生成的每一段都是一个连贯的选择，而不是每个路点各自取平均。这正是 Diffusion Policy 生成动作块（action chunk）的方式，只是它的去噪网络还要以相机图像和本体感觉为条件（22.2 节）。

下图的示教从左边的起点出发，绕过中间的障碍物到右边：一部分从上方绕，其余从下方绕，每条路线都带一点平滑的随机扰动（\(\sigma=0.1\) 的平滑高斯过程，所以这个 20 维的分布仍然是两个高斯的混合，score 可以精确算出）。把均方误差回归会给出的条件均值 \(\hl{mean}{\mathbb E[\mathbf a]}\) 也画出来作比较。这张图只有一个固定的场景，用的是精确的 score，要看的是：对整段动作一起去噪时，每一段都完整地选定一边，而且两边的频率与示教一致。19.7 节的 FIG 5 会在一个类似的避障场景里补上这里没有的三样东西：一个现场训练的网络、以障碍物位置为条件的动作头，以及积分步数减到 1 时会发生什么。

So far the "data" were 2D points. For a robot, the object being generated becomes a whole chunk of actions: the positions of the next 10 waypoints, say, a 20-dimensional vector. A diffusion model does not care about the dimension; the forward process, the training objective and the samplers do not change by a single symbol. It denoises the whole chunk at once, so each chunk it produces is one coherent choice rather than an average taken waypoint by waypoint. This is exactly how Diffusion Policy generates action chunks, except that its denoising network is also conditioned on camera images and proprioception (Lesson 22.2).

In the figure below the demonstrations start on the left, get around the obstacle in the middle and end on the right: some pass above it and the rest below, each route with a little smooth random perturbation (a smooth Gaussian process with \(\sigma=0.1\), so the 20-dimensional distribution is still a mixture of two Gaussians and its score can be computed exactly). For comparison, the conditional mean \(\hl{mean}{\mathbb E[\mathbf a]}\) that squared-error regression would give is drawn as well. This figure has one fixed scene and uses the exact score; what it shows is that denoising the whole chunk at once makes every chunk commit fully to one side, with the two sides at the frequencies of the demonstrations. FIG 5 of Lesson 19.7 returns to a similar obstacle scene and adds the three things missing here: a network trained live, an action head conditioned on the obstacle's position, and what happens when the number of integration steps drops to 1.