Energy-based 模型Energy-Based Models

很多机器人问题天然是“打分”问题，而不是“给出答案”的问题。一个抓取好不好、一个动作和当前观测配不配、一段轨迹有没有撞上东西，判断起来都比直接写出“最好的那个”容易得多。Energy-based 模型就把这种判断本身当作模型：一个网络 \(E_\theta(x)\) 给每个候选 \(x\) 打一个实数分，分数越低越好（沿用物理里“能量越低越稳定”的说法）。要用它的时候，要么找能量最低的 \(x\)（优化），要么按 \(e^{-E}\) 的比例去采样（生成）。

这种写法有三个对机器人特别有用的性质。第一，多峰与跳变：同一个杯子可以从左边抓也可以从右边抓，能量地形上就是两个谷，不会像均方误差那样把两种答案平均成抓在空气里（19.2 节的开头讲过这个问题）；观测稍有变化、最佳动作就从左跳到右，能量函数本身仍然是光滑的，只是谷底换了一个。第二，可组合：“抓得稳”的能量加上“够得着”的能量再加上“不撞桌子”的能量，就是同时满足三个条件的能量，每一项可以分开学、分开换。第三，结构自由：网络只需要输出一个数，不需要像 normalizing flow（19.3 节）那样可逆，也不需要像自回归模型（19.4 节）那样按顺序分解。

自由是有代价的。把能量变成概率要除以一个积分 \(Z=\int e^{-E(x)}dx\)，在高维空间里没人算得出它；训练和采样都得绕开它，而绕开它的工具就是 4.9 节的 MCMC，以及本节的 Langevin 动力学。理解这一点，也就理解了为什么扩散模型（19.6 节）要转而学能量的梯度。2021 年的隐式行为克隆（Implicit BC，Florence 等）正是用 EBM 作为机器人策略，第 7 节会实际训练一个。

Many robot problems are naturally "scoring" problems rather than "produce the answer" problems. Whether a grasp is good, whether an action fits the current observation, whether a trajectory hits something: judging these is much easier than writing down "the best one" directly. An energy-based model takes that judgement itself as the model: a network \(E_\theta(x)\) gives each candidate \(x\) a real-valued score, and lower is better (borrowing the physics phrase "lower energy is more stable"). To use it, you either look for the \(x\) of lowest energy (optimization), or sample in proportion to \(e^{-E}\) (generation).

This formulation has three properties that are especially useful for robots. First, multiple modes and jumps: the same mug can be grasped from the left or from the right, which in the energy landscape is simply two valleys, and nothing averages the two answers into a grasp in mid-air the way mean squared error does (the problem at the start of Lesson 19.2); when a small change in the observation makes the best action jump from left to right, the energy function itself stays smooth, only the lowest valley changes. Second, composability: the energy for "stable grasp" plus the energy for "reachable" plus the energy for "does not hit the table" is the energy for satisfying all three, and each term can be learned and swapped separately. Third, free architecture: the network outputs a single number; it need not be invertible like a normalizing flow (Lesson 19.3) or factorize in order like an autoregressive model (Lesson 19.4).

The freedom has a price. Turning energy into probability means dividing by an integral \(Z=\int e^{-E(x)}dx\), which nobody can compute in high dimensions; both training and sampling must work around it, and the tools for that are the MCMC of Lesson 4.9 and the Langevin dynamics of this lesson. Understanding this also explains why diffusion models (Lesson 19.6) learn the gradient of the energy instead. Implicit behavioural cloning (Implicit BC; Florence et al., 2021) uses an EBM as a robot policy, and Section 7 trains one.

把能量 \(\hl{E}{E(x)}\) 想成一片地形，把概率想成下雨后积在低处的水。能量和概率之间的换算是 Boltzmann 分布（也叫 Gibbs 分布）：

Think of the energy \(\hl{E}{E(x)}\) as a terrain, and of probability as the water that collects in the low places after rain. The conversion between energy and probability is the Boltzmann distribution (also called the Gibbs distribution):

指数保证概率为正，再低的谷也不会变成负概率；除以 \(Z\) 让总概率为 1。温度 \(T\) 决定地形有多“陡”：\(T\) 小，水几乎全挤进最低的那个谷；\(T\) 大，水漫开，各处差不多。两点的概率比只取决于能量差，\(p(x_1)/p(x_2)=e^{-(E(x_1)-E(x_2))/T}\)，与 \(Z\) 无关；整片地形一起抬高一个常数，概率一点也不变，只有 \(Z\) 跟着变。6.11 节的 MPPI 给每条采样轨迹的权重 \(\exp(-f/\lambda)\) 就是这个分布，\(\lambda\) 就是温度。

下图的横轴是夹爪绕竖直轴的偏航角 \(x\)，被关节限位限制在 \([-3,3]\) rad 内（区间有界，所以 \(Z\) 总是有限的）。两个谷是两种都能抓牢的朝向。拖动琥珀色的控制点重塑地形，滑块改温度。

The exponential keeps probabilities positive, however deep a valley is; dividing by \(Z\) makes the total probability 1. The temperature \(T\) sets how "steep" the terrain is: at small \(T\) nearly all the water squeezes into the lowest valley; at large \(T\) it spreads out and every place gets about the same. The probability ratio of two points depends only on the energy difference, \(p(x_1)/p(x_2)=e^{-(E(x_1)-E(x_2))/T}\), independent of \(Z\); raise the whole terrain by a constant and the probabilities do not change at all, only \(Z\) does. The weight \(\exp(-f/\lambda)\) that MPPI in Lesson 6.11 gives each sampled trajectory is exactly this distribution, with \(\lambda\) as the temperature.

In the figure below the horizontal axis is the gripper's yaw angle \(x\) about the vertical axis, limited by the joint to \([-3,3]\) rad (the interval is bounded, so \(Z\) is always finite). The two valleys are two orientations that both give a firm grasp. Drag the amber control points to reshape the terrain; the slider sets the temperature.

一维时可以把整条曲线算出来再积分；二维以上，网格的格子数随维数指数增长，只能采样。4.9 节的 Metropolis 算法只用到能量差，可以直接用在 EBM 上，但它的提议是盲目的随机步。能量有梯度（3.2 节），梯度指向能量升高最快的方向，所以更聪明的做法是先朝 \(-\nabla E\) 走一小步，再加一点高斯噪声。这就是 Langevin 动力学（未校正的 Langevin 算法，ULA）：

In 1D the whole curve can be computed and integrated; from 2D on, the number of grid cells grows exponentially with dimension, and we have to sample. The Metropolis algorithm of Lesson 4.9 uses only energy differences and works directly on an EBM, but its proposals are blind random steps. The energy has a gradient (Lesson 3.2), which points where the energy rises fastest, so a smarter move is a small step along \(-\nabla E\) followed by a little Gaussian noise. That is Langevin dynamics (the unadjusted Langevin algorithm, ULA):

只有前一项就是 6.3 节的梯度下降，所有点都滚进最近的谷底停下来；只有后一项就是随机游走，到处乱跑。两项按这个比例（噪声方差是步长的 2 倍）放在一起，步长 \(\eta\to0\) 时，点的长期分布恰好是 \(p\propto e^{-E}\)（第 6 节说明原因）。注意 \(\nabla_xE\) 里没有 \(Z\)：常数的梯度是零。下图的能量都是已知的解析函数，所以“真实答案”可以用网格精确算出来，用来检验采样器。

The first term alone is the gradient descent of Lesson 6.3: every point rolls into the nearest valley and stops. The second alone is a random walk that wanders everywhere. Put together in this proportion (noise variance twice the step size), the long-run distribution of the points is exactly \(p\propto e^{-E}\) as the step \(\eta\to0\) (Section 6 explains why). Note that \(\nabla_xE\) contains no \(Z\): the gradient of a constant is zero. The energies in the figure below are known analytic functions, so the "true answer" can be computed exactly on a grid and used to check the sampler.

现在能量不再是给定的，而是一个网络 \(E_\theta(x)\)，要从数据里学。最自然的目标是 4.6 节的最大似然：让数据的 \(\log p_\theta(x)=-E_\theta(x)-\log Z(\theta)\) 尽量大。第一项好办，第二项里的 \(Z\) 算不出来，但它的梯度可以写成一个期望（第 6 节推导）：

Now the energy is no longer given: it is a network \(E_\theta(x)\) to be learned from data. The most natural objective is the maximum likelihood of Lesson 4.6: make the data's \(\log p_\theta(x)=-E_\theta(x)-\log Z(\theta)\) as large as possible. The first term is easy; the \(Z\) in the second cannot be computed, but its gradient can be written as an expectation (derived in Section 6):

读法很直白：在数据点上把能量压低，在模型自己的样本上把能量抬高。两者一样多的时候，也就是模型的样本分布和数据分布一致时，梯度为零。第一项叫“正相”，用一批数据就能估计；第二项叫“负相”，需要从当前模型里采样，而这正是上一节的 Langevin。每走一步参数就做一次长链 MCMC 太贵了，Hinton（2002）的对比散度（CD）只跑 \(K\) 步短链；Tieleman（2008）的持续对比散度（PCD）让链在参数更新之间不重置，接着上一轮的位置继续走，因为参数每步只变一点，上一轮的样本已经接近当前模型的分布。下图用 PCD 训练一个 2→32→32→1 的 MLP（GELU 激活）：一个容量 1024 的样本缓冲区，每次取 32 条链跑 \(K\) 步 Langevin（\(\eta=0.006\)），其中 5% 从均匀噪声重新开始；损失再加一项 \(0.1\,(E^2_{\text{data}}+E^2_{\text{neg}})\)，防止能量整体漂移（Du 与 Mordatch，2019 的做法）。

It reads plainly: push the energy down on data points, push it up on the model's own samples. When the two balance, that is when the model's samples are distributed like the data, the gradient is zero. The first term is the "positive phase", estimated with a batch of data; the second is the "negative phase", which needs samples from the current model, and that is exactly the Langevin of the previous section. Running a long MCMC chain for every parameter update is too expensive, so the contrastive divergence (CD) of Hinton (2002) runs only \(K\) short steps; the persistent contrastive divergence (PCD) of Tieleman (2008) does not reset the chains between parameter updates but continues from where they were, because the parameters change only a little per step and the old samples are already close to the current model's distribution. The figure trains a 2→32→32→1 MLP (GELU activations) with PCD: a buffer of 1024 samples, from which each step takes 32 chains and runs \(K\) Langevin steps (\(\eta=0.006\)), 5% of them restarted from uniform noise; the loss also has a term \(0.1\,(E^2_{\text{data}}+E^2_{\text{neg}})\) that stops the energies drifting as a whole (the recipe of Du and Mordatch, 2019).

继续训练下去，大簇的质量会在 0.75 上下摆动（默认种子在 1000 到 2000 步之间约为 0.74 到 0.92），NLL 也在 0.5 到 0.6 之间起伏：负相只用了 32 个样本，PCD 的梯度噪声很大，实际训练会逐渐减小学习率。也请看右侧读数里的 \(\bar E_{\text{data}}\) 与 \(\bar E_{\text{neg}}\)：它们是损失的两项，差值时正时负，不是进度条，真正的进度要看 NLL（这里是因为二维才能精确算出 \(\log Z\)；高维时只能看样本）。“链”切到“每次从噪声重来”就是不持续的短链 CD：在这个二维例子里它也能学到还不错的能量（默认种子下，800 到 1200 步之间 NLL 在 1.05 上下，持续链在 1.0 上下），但只有约一半的负样本到达圆环附近（持续链超过 90%），其余停在半路；Nijkamp 等（2019）研究过这种“不收敛的短链”学到的到底是什么。

Train on and the big cluster's mass swings around 0.75 (about 0.74 to 0.92 between steps 1000 and 2000 with the default seed) while the NLL wanders between 0.5 and 0.6: the negative phase uses only 32 samples, so the PCD gradient is very noisy, and real training gradually lowers the learning rate. Look too at \(\bar E_{\text{data}}\) and \(\bar E_{\text{neg}}\) in the readout: they are the two terms of the loss, their difference flips sign, and they are not a progress bar; real progress is the NLL (available here only because \(\log Z\) can be computed exactly in 2D; in high dimensions you can only look at samples). Switching "chains" to "restart from noise" gives non-persistent short-run CD: in this 2D example it also learns a fairly decent energy (with the default seed, between steps 800 and 1200 its NLL hovers around 1.05, against about 1.0 for persistent chains), but only about half of the negatives get near the ring (over 90% with persistent chains), the rest stopping halfway; Nijkamp et al. (2019) studied what such "non-convergent short-run" chains actually learn.

两个能量相加，对应的概率相乘：\(e^{-(E_1+E_2)}=e^{-E_1}e^{-E_2}\)。概率相乘就是“与”：只有两个模型都认为合理的地方，乘积才大。Hinton 把这叫作专家乘积（product of experts）。反过来，对两个概率求和是“或”，在能量上就是 \(-\log(e^{-E_1}+e^{-E_2})\)。显式的采样器（GAN 的生成器、VAE 的解码器）很难这样拼装：两个生成器的样本没法“相乘”；可两个能量相加之后，同一个 Langevin 采样器照样能用，因为它只需要总能量的梯度。

下图是一个抓取的例子：\(\hl{ring}{E_{\text{grasp}}}\) 是绕杯子一圈的抓取位置（圆环能量，半径 0.7），\(\hl{reach}{E_{\text{box}}}\) 说的是杯子右侧有个箱子挡着，\(x_1>-0.55\) 的位置会撞上（一个光滑的“台阶”：\(\operatorname{softplus}(8(x_1+0.55))\)）。总能量是 \(E=E_{\text{grasp}}+\lambda E_{\text{box}}\)，两者从没一起训练过。

Adding two energies multiplies the corresponding probabilities: \(e^{-(E_1+E_2)}=e^{-E_1}e^{-E_2}\). Multiplying probabilities is an "AND": the product is large only where both models find things plausible. Hinton called this a product of experts. Conversely, summing two probabilities is an "OR", which in energies is \(-\log(e^{-E_1}+e^{-E_2})\). Explicit samplers (a GAN's generator, a VAE's decoder) are hard to assemble like this: there is no way to "multiply" the samples of two generators; but once two energies are added, the same Langevin sampler still works, because it only needs the gradient of the total energy.

The figure is a grasping example: \(\hl{ring}{E_{\text{grasp}}}\) is the grasp positions around a mug (a ring energy of radius 0.7), and \(\hl{reach}{E_{\text{box}}}\) says that a box stands to the right of the mug, so positions with \(x_1>-0.55\) collide with it (a smooth "step": \(\operatorname{softplus}(8(x_1+0.55))\)). The total energy is \(E=E_{\text{grasp}}+\lambda E_{\text{box}}\); the two terms were never trained together.

EBM 由一个参数化的能量 \(E_\theta:\mathcal X\to\mathbb R\) 定义：

An EBM is defined by a parametrized energy \(E_\theta:\mathcal X\to\mathbb R\):

\(Z\) 在统计物理里叫配分函数。只要积分有限（例如 \(\mathcal X\) 有界、\(E_\theta\) 连续，或者 \(E_\theta\) 在远处增长得够快），这就是一个合法的密度。温度只是把能量除以 \(T\)，可以吸收进 \(E_\theta\) 里，所以学习时通常取 \(T=1\)。离散的 \(x\)（类别、token）积分换成求和；类别少时 \(Z\) 直接算得出来，这就是 softmax（16.1 节）。

In statistical physics \(Z\) is called the partition function. As long as the integral is finite (for example \(\mathcal X\) bounded and \(E_\theta\) continuous, or \(E_\theta\) growing fast enough far away), this is a valid density. Temperature merely divides the energy by \(T\) and can be absorbed into \(E_\theta\), so for learning one usually takes \(T=1\). For discrete \(x\) (classes, tokens) the integral becomes a sum; with few classes \(Z\) can be computed outright, and that is the softmax (Lesson 16.1).

对 \(\log Z\) 求导，把导数移进积分（2.2 节的链式法则）：

Differentiate \(\log Z\) and move the derivative inside the integral (the chain rule of Lesson 2.2):

代入 \(-\log p_\theta(x)=E_\theta(x)+\log Z(\theta)\) 并对数据取平均，就得到第 4 节的 \(\hl{pos}{\mathbb E_{p_{\text{data}}}[\nabla_\theta E_\theta]}-\hl{neg}{\mathbb E_{p_\theta}[\nabla_\theta E_\theta]}\)。两点值得记住。第一，整个困难被集中到“从 \(p_\theta\) 采样”这一件事上；样本越不准，负相的估计偏差越大。第二，实现上常写一个“替身损失” \(\frac1B\sum_i E_\theta(x_i)-\frac1B\sum_i E_\theta(x'_i)\)，其中负样本 \(x'_i\) 被当作常数（不对它反向传播），它对 \(\theta\) 的梯度就是上式的估计。这个替身损失的数值没有意义：它可以是任何正负数，FIG 3 的读数显示了这一点。

Substituting \(-\log p_\theta(x)=E_\theta(x)+\log Z(\theta)\) and averaging over the data gives Section 4's \(\hl{pos}{\mathbb E_{p_{\text{data}}}[\nabla_\theta E_\theta]}-\hl{neg}{\mathbb E_{p_\theta}[\nabla_\theta E_\theta]}\). Two things are worth remembering. First, the entire difficulty is concentrated in one task: "sample from \(p_\theta\)"; the worse the samples, the more biased the negative phase. Second, implementations usually write a "surrogate loss" \(\frac1B\sum_i E_\theta(x_i)-\frac1B\sum_i E_\theta(x'_i)\) in which the negatives \(x'_i\) are treated as constants (no backpropagation through them); its gradient with respect to \(\theta\) is an estimate of the expression above. The value of this surrogate loss means nothing: it can be any positive or negative number, as FIG 3's readout shows.

离散更新 \(x_{k+1}=x_k-\eta\nabla E+\sqrt{2\eta}\,\boldsymbol\xi\) 是随机微分方程 \(dx=-\nabla E(x)\,dt+\sqrt2\,dW\) 的欧拉–丸山离散化（时间步 \(\eta\)，对照 2.5 节的欧拉法）。这个方程的密度 \(\rho(x,t)\) 服从 Fokker–Planck 方程

The discrete update \(x_{k+1}=x_k-\eta\nabla E+\sqrt{2\eta}\,\boldsymbol\xi\) is the Euler–Maruyama discretization (time step \(\eta\); compare Euler's method in Lesson 2.5) of the stochastic differential equation \(dx=-\nabla E(x)\,dt+\sqrt2\,dW\). Its density \(\rho(x,t)\) obeys the Fokker–Planck equation

当 \(\rho\propto e^{-E}\) 时 \(E+\log\rho\) 是常数，右边为零，所以 \(p\) 是平稳分布：漂移把概率往谷底推，扩散把它往外摊，两者恰好平衡。噪声系数必须是 \(\sqrt{2\eta}\)：换成别的倍数，平稳分布就变成别的温度下的 \(e^{-E/T}\)。很多 EBM 实现（包括 Du 与 Mordatch 2019）把步长和噪声分开调，噪声远小于 \(\sqrt{2\eta}\)，那就已经不是在 \(T=1\) 下采样了，更像是带一点抖动的梯度下降。

有限步长的偏差：在一维二次谷 \(E=\tfrac c2x^2\) 里，更新是 \(x_{k+1}=(1-\eta c)x_k+\sqrt{2\eta}\,\xi\)，平稳方差 \(v\) 满足 \(v=(1-\eta c)^2v+2\eta\)，解得 \(v=\frac1c\cdot\frac{1}{1-\eta c/2}\)，而精确值是 \(1/c\)。\(\eta c\to2\) 时方差发散（6.3 节梯度下降的稳定界）。FIG 2 的圆环曲率 \(c=1/0.15^2\approx44\)：\(\eta=0.035\) 时 \(\eta c\approx1.56\)，标准差约是精确值的 2.1 倍，图中测得约 2 倍。

MALA（Metropolis 校正的 Langevin，Roberts 与 Tweedie，1996）把 Langevin 的一步当作 4.9 节 Metropolis–Hastings 的提议 \(q(x'|x)=\mathcal N\big(x'; x-\eta\nabla E(x),\,2\eta I\big)\)，以概率

When \(\rho\propto e^{-E}\), \(E+\log\rho\) is constant and the right-hand side vanishes, so \(p\) is the stationary distribution: the drift pushes probability towards the valley bottoms, the diffusion spreads it out, and the two balance exactly. The noise coefficient must be \(\sqrt{2\eta}\): with another multiple, the stationary distribution becomes \(e^{-E/T}\) at some other temperature. Many EBM implementations (including Du and Mordatch 2019) tune the step and the noise separately, with noise far below \(\sqrt{2\eta}\); that is no longer sampling at \(T=1\), but closer to gradient descent with a little jitter.

Finite-step bias: in a 1D quadratic valley \(E=\tfrac c2x^2\) the update is \(x_{k+1}=(1-\eta c)x_k+\sqrt{2\eta}\,\xi\), and the stationary variance \(v\) satisfies \(v=(1-\eta c)^2v+2\eta\), so \(v=\frac1c\cdot\frac{1}{1-\eta c/2}\), whereas the exact value is \(1/c\). As \(\eta c\to2\) the variance blows up (the stability limit of gradient descent in Lesson 6.3). FIG 2's ring has curvature \(c=1/0.15^2\approx44\): at \(\eta=0.035\), \(\eta c\approx1.56\) and the standard deviation should be about 2.1 times the exact one; the figure measures about 2.

MALA (the Metropolis-adjusted Langevin algorithm; Roberts and Tweedie, 1996) treats a Langevin step as the proposal \(q(x'|x)=\mathcal N\big(x'; x-\eta\nabla E(x),\,2\eta I\big)\) of the Metropolis–Hastings algorithm of Lesson 4.9 and accepts it with probability

满足细致平衡，所以对任何步长都没有偏差；\(Z\) 在比值里消掉了。步长大了接受率下降，链走得慢；FIG 2 的圆环在 \(\eta=0.035\) 时接受率约 0.6。

This satisfies detailed balance, so there is no bias at any step size; \(Z\) cancels in the ratio. Larger steps lower the acceptance rate and slow the chain; on FIG 2's ring at \(\eta=0.035\) the acceptance rate is about 0.6.

Score matching（Hyvärinen，2005）比较的是梯度而不是密度：最小化 \(\tfrac12\mathbb E_{p_{\text{data}}}\|\nabla_x\log p_\theta-\nabla_x\log p_{\text{data}}\|^2\)，\(\nabla_x\log p_\theta=-\nabla_xE_\theta\) 与 \(Z\) 无关；分部积分后未知的 \(\nabla_x\log p_{\text{data}}\) 也能消掉。它的去噪版本是 19.6 节扩散模型的训练目标。噪声对比估计（NCE，Gutmann 与 Hyvärinen，2010）训练一个分类器区分数据和一个已知的噪声分布 \(q\)，把 \(\log Z\) 当作一个可学的参数；数据与噪声样本一样多时，分类器的最优 logit 恰是 \(\log p_{\text{data}}-\log q\)（19.2 节的密度比技巧）。第 7 节的 IBC 用的是它的多类版本 InfoNCE（18.3 节的对比学习也用它）：对每个数据对 \((o,a)\) 采 \(N\) 个反例动作 \(\tilde a_j\)，

Score matching (Hyvärinen, 2005) compares gradients instead of densities: minimize \(\tfrac12\mathbb E_{p_{\text{data}}}\|\nabla_x\log p_\theta-\nabla_x\log p_{\text{data}}\|^2\), where \(\nabla_x\log p_\theta=-\nabla_xE_\theta\) does not involve \(Z\); integrating by parts also removes the unknown \(\nabla_x\log p_{\text{data}}\). Its denoising version is the training objective of the diffusion models of Lesson 19.6. Noise-contrastive estimation (NCE; Gutmann and Hyvärinen, 2010) trains a classifier to tell data from a known noise distribution \(q\), treating \(\log Z\) as a learnable parameter; with equal numbers of data and noise samples, the optimal logit of the classifier is exactly \(\log p_{\text{data}}-\log q\) (the density-ratio trick of Lesson 19.2). Section 7's IBC uses its multi-class version InfoNCE (also used by the contrastive learning of Lesson 18.3): for each data pair \((o,a)\), draw \(N\) counter-example actions \(\tilde a_j\) and minimize

这就是一个 \(N+1\) 类的 softmax 交叉熵，正确答案是示教动作。反例从动作范围内均匀采样时，分母是对 \(Z(o)=\int e^{-E_\theta(o,a)}da\) 的蒙特卡洛估计（差一个常数因子），\(N\) 越大越准。动作维数高时，均匀反例绝大多数落在显然很差的地方，提供的信息很少，需要的 \(N\) 随维数急剧增长。

This is just an \((N+1)\)-way softmax cross-entropy whose correct answer is the demonstrated action. With counter-examples drawn uniformly over the action range, the denominator is a Monte Carlo estimate of \(Z(o)=\int e^{-E_\theta(o,a)}da\) (up to a constant factor), more accurate for larger \(N\). When the action has many dimensions, nearly all uniform counter-examples land in obviously bad places and carry little information, so the \(N\) required grows steeply with dimension.

乘积 \(p_1(x)p_2(x)/Z_{12}\) 的能量是 \(\hl{ring}{E_1}+\hl{reach}{E_2}\)（常数不计）；混合 \(\pi_1p_1+\pi_2p_2\) 的能量是 \(-\log\big(\pi_1e^{-E_1}/Z_1+\pi_2e^{-E_2}/Z_2\big)\)。注意“或”需要知道每个分量的 \(Z_i\)（或者接受由它们决定的比例）：FIG 4 的两个圆环按周长归一化，正是为了让两个杯子各得一半。“与”则不需要任何 \(Z\)，这是 EBM 组合最常用的形式。条件模型 \(E_\theta(o,a)\) 也是同一个思路：观测 \(o\) 固定时，它是动作上的一个能量。

The product \(p_1(x)p_2(x)/Z_{12}\) has energy \(\hl{ring}{E_1}+\hl{reach}{E_2}\) (up to a constant); the mixture \(\pi_1p_1+\pi_2p_2\) has energy \(-\log\big(\pi_1e^{-E_1}/Z_1+\pi_2e^{-E_2}/Z_2\big)\). Note that the OR needs each component's \(Z_i\) (or accepts the proportions they imply): FIG 4 normalizes its two rings by their circumferences precisely so that each mug gets half. The AND needs no \(Z\) at all, which is why it is the most common way to compose EBMs. A conditional model \(E_\theta(o,a)\) follows the same idea: with the observation \(o\) fixed, it is an energy over actions.

普通的行为克隆（22.1 节）学一个显式策略 \(\hat a=\pi_\theta(o)\)，用均方误差回归示教动作。Florence 等（CoRL 2021）的隐式行为克隆（Implicit BC，IBC）改学一个能量 \(E_\theta(o,a)\)，策略是

Ordinary behavioural cloning (Lesson 22.1) learns an explicit policy \(\hat a=\pi_\theta(o)\) by regressing the demonstrated actions with mean squared error. The implicit behavioural cloning (Implicit BC, IBC) of Florence et al. (CoRL 2021) learns an energy \(E_\theta(o,a)\) instead, and the policy is

用上一节的 InfoNCE 训练，反例在动作范围内采样。推理时没有梯度下降可以“一步到位”，原文给了几种做法：无梯度优化（DFO：撒一批动作，按 \(\operatorname{softmax}(-E)\) 重采样，加上逐步缩小的噪声，重复几轮，与 6.11 节的 CEM 同一路数）、维数更高时逐维进行的自回归版本，以及 Langevin MCMC。下面是一个推冰球的例子：观测 \(o\) 是冰球相对前方立柱的横向偏移，动作 \(a\) 是推杆经过立柱时的横向位置。冰球明显偏左（\(o<-0.2\)）时示教者总从左边绕，\(a=-0.55+0.25\,o\)；明显偏右（\(o>0.2\)）时总从右边绕，\(a=0.55+0.25\,o\)；中间那一段左右各一半。这既有多峰（中间段），又有跳变（\(o=\pm0.2\) 处）。同时训练两个模型：显式的 1→32→32→1 MLP（MSE），和隐式的 \(E_\theta(o,a)\)，一个 2→32→32→1 MLP（InfoNCE，每个示教 16 个均匀反例）。

It is trained with the InfoNCE of the previous section, with counter-examples sampled over the action range. At inference there is no single gradient step that "gets there", and the paper gives several options: derivative-free optimization (DFO: scatter a batch of actions, resample them according to \(\operatorname{softmax}(-E)\), add noise that shrinks every round, and repeat a few times, in the same spirit as the CEM of Lesson 6.11), an autoregressive version that goes one dimension at a time for higher-dimensional actions, and Langevin MCMC. Below is a puck-pushing example: the observation \(o\) is the puck's lateral offset from a post ahead of it, and the action \(a\) is the pusher's lateral position as it passes the post. When the puck is clearly to the left (\(o<-0.2\)) the demonstrator always passes on the left, \(a=-0.55+0.25\,o\); clearly to the right (\(o>0.2\)), always on the right, \(a=0.55+0.25\,o\); in the middle band, half and half. That has both multiple modes (the middle band) and jumps (at \(o=\pm0.2\)). Two models train side by side: an explicit 1→32→32→1 MLP (MSE) and an implicit \(E_\theta(o,a)\), a 2→32→32→1 MLP (InfoNCE, 16 uniform counter-examples per demonstration).

为什么 \(\arg\min\) 能跳变？能量 \(E_\theta(o,a)\) 是光滑的，但“哪个谷更低”可以随 \(o\) 突然换边：两个谷的深度连续地此消彼长，交叉的那一刻最低点从一个谷跳到另一个谷。显式网络要表示跳变，只能把一个连续函数掰得很陡；而在多峰的中间段，MSE 的最优解是条件均值，正好落在两个模式之间的空地（这里就是撞上立柱）。IBC 论文把这一点作为主要论据，并在它的实验里报告：隐式策略总体上优于 MSE 和混合密度网络（MDN）这类常见的显式策略。

Why can the \(\arg\min\) jump? The energy \(E_\theta(o,a)\) is smooth, but "which valley is lower" can switch sides abruptly as \(o\) changes: the depths of the two valleys trade off continuously, and at the moment they cross the minimum jumps from one valley to the other. An explicit network can represent a jump only by bending a continuous function very steeply; and in the multimodal middle band the MSE optimum is the conditional mean, which lands exactly in the empty space between the two modes (here: straight into the post). The IBC paper makes this its central argument, and in its experiments reports that implicit policies generally outperform common explicit ones such as MSE and mixture density networks (MDN).