ACT 与 Diffusion PolicyACT & Diffusion Policy

22.1 节把模仿学习写成监督学习：收集示教 \((\mathbf o_t,\mathbf a_t)\)，训练一个策略 \(\pi_\theta(\mathbf a\mid\mathbf o)\) 去复现示教者的动作。这在仿真和简单任务上很好用，可一到真实的精细操作（把电池塞进卡槽、拉开自封袋、给鞋穿鞋带），朴素的行为克隆常常一败涂地。原因主要有两个。

一、示教是多模态的。同一个场景，人有时从左边绕，有时从右边绕；有时先抓杯沿，有时先抓杯柄。用均方误差 (MSE) 回归，网络学到的是条件均值（16.2 节），两个都对的动作的平均却可能哪个都不对：左绕和右绕的平均就是直直撞上去。二、误差会累积。真实机器人常以 50 Hz 控制，20 秒就是 1000 次决策；每一步的小偏差把机器人带到示教里没见过的状态，后面的预测更差（22.1 节的 \(O(\varepsilon T^2)\) 界）。人类示教还有停顿、犹豫这类“非马尔可夫”的行为：同一个状态，下一步有时动、有时不动，只看当前观测的策略很难学。

2023 年的两篇工作给出了后来成为默认做法的答案。ACT（Action Chunking with Transformers，Zhao、Kumar、Levine、Finn，RSS 2023，配合低成本双臂遥操作系统 ALOHA，22.3 节）一次预测接下来 \(k\) 步的整段动作（动作分块），把有效决策次数缩短为原来的 \(1/k\)，用条件 VAE 处理示教的多样性，用时间集成把重叠的预测平滑地拼起来。Diffusion Policy（Chi 等，RSS 2023）把策略写成一个以观测为条件的扩散模型：从噪声出发，对一整段动作反复去噪，天然能表示多峰分布，再按“预测一段、执行一部分、重新预测”的滚动时域方式执行。两者都已成为模仿学习的标准基线，Hugging Face 的开源库 LeRobot 里两者都有实现。

本节用的零件你都见过：条件 VAE（19.1 节）、带查询向量的交叉注意力（17.3 节）、DDPM 的训练与采样（19.6 节）、flow matching（19.7 节）。这里不再推导它们，而是看它们怎样组装成机器人策略，以及每个设计选择解决了哪个具体问题。

Lesson 22.1 cast imitation learning as supervised learning: collect demonstrations \((\mathbf o_t,\mathbf a_t)\) and train a policy \(\pi_\theta(\mathbf a\mid\mathbf o)\) to reproduce the demonstrator's actions. That works well in simulation and on simple tasks, but on real fine manipulation (slotting a battery, opening a ziploc bag, putting on a shoe) naive behaviour cloning often fails badly. There are two main reasons.

One: demonstrations are multimodal. In the same scene a person sometimes goes around on the left and sometimes on the right, sometimes grasps a cup by the rim and sometimes by the handle. Regressing with a mean squared error (MSE) teaches the network the conditional mean (Lesson 16.2), and the average of two correct actions may be neither: the average of left and right is straight into the obstacle. Two: errors compound. Real robots are often controlled at 50 Hz, so 20 seconds is 1000 decisions; each small deviation carries the robot into states the demonstrations never showed, where the next prediction is worse (the \(O(\varepsilon T^2)\) bound of Lesson 22.1). Human demonstrations also contain pauses and hesitation, "non-Markovian" behaviour: in the same state the next step is sometimes a move and sometimes not, which a policy that sees only the current observation struggles to learn.

Two works from 2023 gave the answers that became the default. ACT (Action Chunking with Transformers; Zhao, Kumar, Levine & Finn, RSS 2023, together with the low-cost bimanual teleoperation system ALOHA, Lesson 22.3) predicts the whole sequence of the next \(k\) actions at once (action chunking), which cuts the number of effective decisions by a factor of \(k\); it handles the variability of the demonstrations with a conditional VAE and blends overlapping predictions with temporal ensembling. Diffusion Policy (Chi et al., RSS 2023) writes the policy as a diffusion model conditioned on the observation: starting from noise it repeatedly denoises a whole action sequence, which represents multimodal distributions naturally, and it executes in a receding horizon: predict a sequence, execute part of it, predict again. Both are now standard baselines of imitation learning, and Hugging Face's open-source library LeRobot implements both.

You have seen every part used here: the conditional VAE (Lesson 19.1), cross-attention with query vectors (Lesson 17.3), DDPM training and sampling (Lesson 19.6), and flow matching (Lesson 19.7). Instead of deriving them again, this lesson shows how they are assembled into robot policies and which concrete problem each design choice solves.

第 19 章有六节都以同一幅图收尾：示教者从障碍物左边或右边绕过，均方误差回归直直撞上，而生成模型绕了过去。它们是条件 VAE（19.1 节 FIG 6）、条件 GAN（19.2 节 FIG 6）、自回归动作 token（19.4 节 FIG 6）、隐式 BC（19.5 节 FIG 5）、扩散（19.6 节 FIG 6）和 flow matching（19.7 节 FIG 5）。22.1 节 FIG 5 又在闭环里说明，策略还得在一段时间内坚持同一种绕法。那些图评判的都是针对一个给定观测、只生成一次的一个动作或一个动作块。本节不再重复“生成模型能绕过去”这一点，而是补上它们没做的那一步：让策略在整条轨迹上闭环运行，边走边重新查询动作块，并看块与块怎样拼接，ACT 用时间集成，Diffusion Policy 用执行时域；再在同一个任务、同一组测试上，把两种已发表的做法和 MSE 基线并排比较。FIG 1 还多画了一样第 19 章没画的东西：中位数，也就是 ACT 的 L1 损失所逼近的量。

本节所有实验共用一个任务。俯视的平面上，夹爪从 \(y=0.1\) 匀速前进到 \(y=1.9\)（40 步），策略只决定横向位置 \(x\)。\(y=1\) 处有一个半径 0.22 m 的障碍物，横向位置 \(o\in[-0.3,0.3]\) 每次不同；夹爪半径按 0.05 m 算，所以中心离障碍物圆心小于 0.27 m 就算撞上。示教者先直行，在 \(y\approx0.55\) 处迅速横移到障碍物左边或右边 0.55 m 处（加 0.05 m 的噪声），绕过后再回到中线。障碍物越靠右越倾向于从左边绕：从左绕的概率是 \(p_L(o)=1/(1+e^{-4o})\)，所以 \(o=0\) 时左右各半，\(o=0.3\) 时也只有 77% 从左绕。共 200 条示教。

把示教在障碍物处的横向位置 \(a\) 看成要预测的量。它的分布是两个窄峰的混合：\(a\approx o-0.55\)（左）的概率是 \(p_L\)，\(a\approx o+0.55\)（右）的概率是 \(1-p_L\)。下面这张图不训练任何网络，直接画出三种“把分布变成一个动作”的方式：

• 均值 \(\hl{mean}{\mathbb E[a\mid o]}\)：均方误差回归的最优解。两个峰的加权平均，相对障碍物的偏移只有 \(0.55\,|1-2p_L|\)。
• 中位数 \(\hl{med}{\operatorname{med}[a\mid o]}\)：L1 回归的最优解（16.2 节）。只要一边的概率超过一半，它就落在多数那一边的峰上。ACT 用的就是 L1 损失。
• 采样：从分布里抽一个 \(a\)。每次落在某一个峰上，左右的比例和示教一致。扩散策略做的就是这件事。

Six lessons of chapter 19 end with the same picture: demonstrators pass an obstacle on the left or on the right, squared-error regression drives straight into it, and a generative model gets around. They are the conditional VAE (FIG 6 of Lesson 19.1), the conditional GAN (FIG 6 of Lesson 19.2), autoregressive action tokens (FIG 6 of Lesson 19.4), implicit BC (FIG 5 of Lesson 19.5), diffusion (FIG 6 of Lesson 19.6) and flow matching (FIG 5 of Lesson 19.7). FIG 5 of Lesson 22.1 then showed in closed loop that a policy must also keep to one way around for a while. Each of those figures judged one action or one chunk, generated once for a given observation. This lesson does not repeat that generative models get around; it adds the step those figures left out: running the policies closed loop over whole episodes, re-querying chunks as the gripper moves, and stitching the chunks together, by temporal ensembling in ACT and by an execution horizon in Diffusion Policy. It then compares the two published recipes side by side with an MSE baseline on one task and one test grid. FIG 1 also draws one thing chapter 19 did not: the median, which is what ACT's L1 loss aims at.

All experiments in this lesson share one task. Seen from above, a gripper advances at constant speed from \(y=0.1\) to \(y=1.9\) (40 steps), and the policy only decides the lateral position \(x\). At \(y=1\) there is an obstacle of radius 0.22 m whose lateral position \(o\in[-0.3,0.3]\) changes every time; with a gripper radius of 0.05 m, the gripper collides when its centre comes within 0.27 m of the obstacle's centre. The demonstrator first goes straight, then at \(y\approx0.55\) quickly sidesteps to 0.55 m left or right of the obstacle (plus 0.05 m of noise), passes it and returns to the centre line. The further right the obstacle, the more often the demonstrator passes on the left: with probability \(p_L(o)=1/(1+e^{-4o})\), so at \(o=0\) it is half and half, and even at \(o=0.3\) only 77% of passes are on the left. There are 200 demonstrations.

Treat the lateral position \(a\) of a demonstration at the obstacle as the quantity to predict. Its distribution is a mixture of two narrow peaks: \(a\approx o-0.55\) (left) with probability \(p_L\) and \(a\approx o+0.55\) (right) with probability \(1-p_L\). The figure below trains no network; it draws three ways of turning that distribution into one action:

• The mean \(\hl{mean}{\mathbb E[a\mid o]}\): the optimum of squared-error regression. A weighted average of the two peaks, whose offset from the obstacle is only \(0.55\,|1-2p_L|\).
• The median \(\hl{med}{\operatorname{med}[a\mid o]}\): the optimum of L1 regression (Lesson 16.2). As soon as one side holds more than half the probability, it sits on the peak of that majority side. ACT uses an L1 loss.
• A sample: draw one \(a\) from the distribution. Each draw lands on one of the peaks, with left and right in the demonstrated proportions. This is what a diffusion policy does.

现在真的训练网络。三种策略看到的观测完全一样：夹爪横向位置 \(x\)、相对障碍物的位置 \(x-o\)、障碍物位置 \(o\)、前进进度 \(u\in[0,1]\) 以及 \(u\) 的 3 对正余弦特征（共 10 维）。三者输出的也是同一个东西：接下来 \(k=8\) 步的横向偏移（一个动作块，相对当前位置，每一维按训练数据的范围归一化到 \([-1,1]\)，Diffusion Policy 也这样做）。唯一的区别是学什么、怎么用：

• MSE 回归（紫色）：10 → 64 → 64 → 8 的 tanh 网络，均方误差。每 8 步查询一次，执行整块。它就是“加了动作分块的行为克隆”，用来说明分块本身治不了多模态。
• ACT 式 CVAE（蓝色）：编码器 \(q_\phi(z\mid\mathbf a_{t:t+k},\mathbf o_t)\)（18 → 64 → 2），解码器 \(\pi_\theta(\mathbf a_{t:t+k}\mid\mathbf o_t,z)\)（11 → 64 → 64 → 8），一维的 \(z\)，L1 重建加 \(\beta=0.6\) 倍 KL。测试时 \(z=0\)，每一步都查询，用时间集成（\(m=0.1\)）合成实际动作。网络是 MLP 而不是 Transformer：这个任务的观测只有 10 个数，用不着注意力；ACT 的 Transformer 是为了融合四路相机的上千个图像 token（第 4 节）。
• 扩散策略（绿色）：一个噪声预测网络 \(\boldsymbol\epsilon_\theta(\mathbf A^k,\mathbf o,k)\)（输入为 8 维带噪动作块、10 维观测和 8 维去噪步编码，26 → 96 → 96 → 8，ReLU），\(K=50\) 步余弦噪声表，权重取指数滑动平均（EMA）。采样用 10 步 DDIM，每 8 步重新采样一块并执行整块。

三个网络每步各取 16 个样本、用 Adam（学习率 0.003）在你的浏览器里同步训练，随机数都有固定种子。每训练 500 步，程序暂停训练，在一个测试网格上评估三种策略：25 个障碍物位置（\(o=-0.3,-0.275,\dots,0.3\)）乘 3 个起点（\(x_0=-0.02,0,0.02\)），共 75 次完整的 40 步执行。

Now we train real networks. All three policies see exactly the same observation: the gripper's lateral position \(x\), its position relative to the obstacle \(x-o\), the obstacle position \(o\), the progress \(u\in[0,1]\) and 3 sine–cosine pairs of \(u\) (10 numbers). All three output the same thing too: the lateral offsets of the next \(k=8\) steps (an action chunk, relative to the current position, each dimension normalized to \([-1,1]\) by its range in the training data, as Diffusion Policy does). The only differences are what they learn and how they are used:

• MSE regression (purple): a 10 → 64 → 64 → 8 tanh network with a squared error. It is queried every 8 steps and the whole chunk is executed. This is "behaviour cloning with action chunking", here to show that chunking by itself does not cure multimodality.
• ACT-style CVAE (blue): an encoder \(q_\phi(z\mid\mathbf a_{t:t+k},\mathbf o_t)\) (18 → 64 → 2), a decoder \(\pi_\theta(\mathbf a_{t:t+k}\mid\mathbf o_t,z)\) (11 → 64 → 64 → 8), a one-dimensional \(z\), and an L1 reconstruction plus \(\beta=0.6\) times the KL. At test time \(z=0\); the policy is queried at every step and temporal ensembling (\(m=0.1\)) forms the executed action. The networks are MLPs, not transformers: this task's observation is 10 numbers and needs no attention; ACT's transformer is there to fuse more than a thousand image tokens from four cameras (Section 4).
• Diffusion policy (green): a noise-prediction network \(\boldsymbol\epsilon_\theta(\mathbf A^k,\mathbf o,k)\) (inputs: the 8-dimensional noisy chunk, the 10-dimensional observation and an 8-dimensional encoding of the denoising step; 26 → 96 → 96 → 8, ReLU), a cosine noise schedule with \(K=50\) steps, and an exponential moving average (EMA) of the weights. Sampling uses 10 DDIM steps; every 8 steps a new chunk is sampled and executed in full.

The three networks train side by side in your browser, 16 samples each per step, with Adam (learning rate 0.003), and every random draw has a fixed seed. Every 500 steps the program pauses training and evaluates all three policies on a test grid: 25 obstacle positions (\(o=-0.3,-0.275,\dots,0.3\)) times 3 starting points (\(x_0=-0.02,0,0.02\)), 75 complete 40-step runs.

三行结果条讲的是三件不同的事。MSE 回归在大部分 \(o\) 上都撞：它学到的整块动作是两种绕法的平均，而且这块动作一旦算出就整段执行，没有机会在中途“掉”进某一边。ACT 式 CVAE 在两侧都成功，失败集中在中间一段：L1 让它在多数一边明显占优时选中多数那一边（FIG 1 的中位数），但真正的中位数在 \(o=0\) 处从一边跳到另一边，而网络是光滑函数，只能用一个斜坡去近似这个跳变，斜坡上的解码结果就落在两峰之间。扩散策略失败得最少，而且失败的位置是分散的：它每次采样都落在某一边，偶尔有一块样本落在两峰之间（训练几千步的小网络还不完美），这些失败没有固定的位置。在默认种子下，3000 到 8000 步之间各次评估的失败率大致是：MSE 回归 45% 到 100%，ACT 15% 到 45%，扩散策略 10% 到 25%（不同的 JavaScript 引擎在数学函数上有极细微的差别，训练中会被放大，所以你看到的个别数字可能差几个百分点）。再训练下去，ACT 和扩散策略的差距会缩小，偶尔有一次评估两者顺序颠倒。

还要注意下面的损失曲线：三条曲线都在平稳下降，可结果条（尤其是 MSE 那一行）在相邻两次评估之间可以变化很大。训练损失和闭环成功率是两回事：损失衡量的是在示教状态上一步的预测误差，成功与否取决于整条轨迹，取决于策略在自己造成的状态上怎么做。机器人学习的论文总是报告真实执行的成功率，原因就在这里。

The three strips tell three different stories. MSE regression collides over most values of \(o\): the whole chunk it learns is the average of the two ways around, and once computed the chunk is executed in full, with no chance to "fall" towards one side halfway. The ACT-style CVAE succeeds on both flanks and fails in one stretch in the middle: L1 makes it choose the majority side when one side clearly dominates (the median of FIG 1), but the true median jumps from one side to the other at \(o=0\), and a network is a smooth function that can only approximate the jump with a ramp, and on the ramp the decoded chunk lies between the peaks. The diffusion policy fails least often, and in scattered places: every sample lands on one side or the other, and occasionally a chunk lands between the peaks (a small network trained for a few thousand steps is not perfect); these failures have no fixed location. With the default seed, the evaluations between 3000 and 8000 steps give failure rates of roughly 45% to 100% for MSE regression, 15% to 45% for ACT and 10% to 25% for the diffusion policy (JavaScript engines differ in tiny details of their maths functions that training amplifies, so a few of your numbers may differ by a few percentage points). With longer training the gap between ACT and the diffusion policy narrows, and once in a while an evaluation ranks them the other way round.

Notice the loss curves too: all three fall steadily, yet the strips (the MSE row above all) can change a lot between two consecutive evaluations. Training loss and closed-loop success are different things: the loss measures one-step prediction error on demonstrated states, while success depends on the whole trajectory, on what the policy does in states it caused itself. That is why robot-learning papers always report success rates of real executions.

ACT 的策略不是 \(\pi(\mathbf a_t\mid\mathbf o_t)\)，而是 \(\pi(\mathbf a_{t:t+k}\mid\mathbf o_t)\)：看一次，说出接下来 \(k\) 步。最朴素的用法是每 \(k\) 步查询一次、整段开环执行，于是一条 \(T\) 步的任务只剩 \(T/k\) 次决策，误差累积的“有效时域”缩短为原来的 \(1/k\)；示教里的停顿也被包在一个块里，不必从单个状态猜“现在该不该动”。代价是两个：块与块的接缝处会跳一下（新观测每 \(k\) 步才进来一次），块内部对意外完全不反应。

ACT 的补救是每一步都查询，于是第 \(t\) 步会收到最多 \(k\) 个预测：\(t\) 时刻做的、\(t-1\) 时刻做的……一直到 \(t-k+1\) 时刻做的。时间集成把它们按 \(\hl{wi}{w_i=e^{-m\,i}}\) 加权平均，其中 \(i=0\) 是最早的那个预测（论文原文：\(w_0\) 是最旧动作的权重），\(m\) 越小，新观测被吸收得越快。注意它平均的是“对同一时刻的不同预测”，而不是相邻时刻的动作，所以不会像普通低通滤波那样把轨迹抹平带偏。

下面用一个一维跟踪任务单独看这件事，没有神经网络：参考轨迹是 \(r(t)=0.25\sin(2\pi t/80)\)，第 63 步时目标突然平移 0.3（比如物体被人挪了一下）。“策略”每次查询都能看到当时为止的参考，但它的预测有误差：每次查询有自己的偏置（标准差 0.03）和随预测步数增长的斜率误差（标准差每步 0.004），就像一个训练得不完美的网络，每次查询犯的错不一样。

ACT's policy is not \(\pi(\mathbf a_t\mid\mathbf o_t)\) but \(\pi(\mathbf a_{t:t+k}\mid\mathbf o_t)\): look once, say the next \(k\) steps. The plainest use is to query every \(k\) steps and execute the chunk open loop, so a \(T\)-step task needs only \(T/k\) decisions and the "effective horizon" over which errors compound shrinks by a factor of \(k\); pauses in the demonstrations are also wrapped inside a chunk, so the policy need not guess from a single state whether to move now. There are two costs: the motion jumps at the seams between chunks (a new observation enters only every \(k\) steps), and inside a chunk nothing reacts to surprises.

ACT's remedy is to query at every step, so step \(t\) receives up to \(k\) predictions: the one made at \(t\), the one made at \(t-1\), and so on back to the one made at \(t-k+1\). Temporal ensembling averages them with weights \(\hl{wi}{w_i=e^{-m\,i}}\), where \(i=0\) is the oldest prediction (in the paper's words, \(w_0\) is the weight of the oldest action), and a smaller \(m\) incorporates new observations faster. Note that it averages "different predictions for the same time step", not the actions of neighbouring steps, so it does not smear and bias the trajectory as an ordinary low-pass filter would.

Below, a one-dimensional tracking task isolates this, with no neural network: the reference is \(r(t)=0.25\sin(2\pi t/80)\), and at step 63 the target suddenly shifts by 0.3 (someone moved the object, say). At each query the "policy" sees the reference up to that moment, but its predictions have errors: each query has its own bias (standard deviation 0.03) and a slope error that grows with the prediction step (standard deviation 0.004 per step), like an imperfectly trained network that errs differently at each query.

逐步查询（\(k=1\) 的极端）反应最快，但每一步都换一个“犯错方式”，执行出来就是抖动。分块执行把每次查询的误差固定在一整块里，块内平滑，代价是接缝跳变和慢反应。时间集成在每一步把 \(k\) 个独立的误差平均掉一部分，所以抖动和接缝都小了很多；它的反应时间在这个例子里主要由 \(k\) 决定（跳变前做的旧预测要等它们的块都过期才完全退出），\(m\) 的影响小得多。\(m\) 太大时最早那个预测占了权重的大头，平均的效果大多消失，又回到了“一块一块”的样子。ACT 论文的消融实验报告，时间集成给 ACT 带来约 3.3% 的成功率提升，给加了分块的 BC-ConvMLP 带来约 4%；对直接检索真实示教动作的最近邻方法 VINN 反而略有下降，作者认为时间集成主要是在平滑参数化模型的预测误差。

Querying every step (the extreme \(k=1\)) reacts fastest, but every step brings a new "way of being wrong", which comes out as jitter. Executing chunks freezes each query's error over a whole chunk: smooth inside, at the price of jumps at the seams and slow reactions. Temporal ensembling averages away part of \(k\) independent errors at every step, so both jitter and seams shrink a lot; in this example its reaction time is set mainly by \(k\) (old predictions made before the jump leave completely only when their chunks expire), and \(m\) matters much less. With too large an \(m\) the oldest prediction dominates the weights, most of the averaging disappears, and the chunk-by-chunk look returns. In ACT's ablations, temporal ensembling improved ACT's success rate by about 3.3% and that of BC-ConvMLP with chunking by about 4%, while it slightly hurt VINN, a nearest-neighbour method that retrieves real demonstrated actions; the authors read this as temporal ensembling mainly smoothing the prediction errors of parametric models.

ACT 把分块策略训练成一个条件 VAE（19.1 节 FIG 6 的同一结构）：编码器只在训练时使用，读入当前关节状态和示教的动作块，输出“风格变量” \(z\) 的高斯分布；解码器（也就是策略）读入观测和 \(z\)，输出动作块。KL 项把 \(z\) 拉向先验 \(\mathcal N(\mathbf 0,I)\)，\(\beta\) 越大，\(z\) 能携带的信息越少。直觉上，\(z\) 应当只编码“光看观测猜不出、要看到示教的动作才知道”的那部分：在我们的任务里，就是这次走左边还是右边。

测试时 ACT 不采样，而是把 \(z\) 固定为先验的均值 0，得到一个确定性的策略。下图用 FIG 2 训练出的 CVAE 来看这个选择意味着什么。

ACT trains its chunked policy as a conditional VAE (the same structure as FIG 6 of Lesson 19.1): the encoder is used only during training; it reads the current joint state and the demonstrated action chunk and outputs a Gaussian over a "style variable" \(z\); the decoder, which is the policy, reads the observation and \(z\) and outputs the chunk. The KL term pulls \(z\) towards the prior \(\mathcal N(\mathbf 0,I)\), and the larger \(\beta\), the less information \(z\) can carry. Intuitively, \(z\) should encode only what cannot be guessed from the observation and is revealed by the demonstrated actions: in our task, whether this pass goes left or right.

At test time ACT does not sample; it fixes \(z\) to the prior mean 0, which gives a deterministic policy. The figure below uses the CVAE trained in FIG 2 to see what that choice means.

编码器确实学会了把左右两种示教分到 \(z\) 的两侧，解码器也确实会在 \(z\) 变号时换边。但 \(z=0\) 落在两者的分界附近：在两边差不多一样可能的那一段 \(o\) 上，它解码出的正是“两峰之间”（分界的确切位置随训练漂移，所以不一定恰好在 \(o=0\)），这就是 FIG 2 里 ACT 的失败段（19.1 节误区 5 讲过同一件事）。那 ACT 为什么还要 CVAE？论文的消融给出了答案：在脚本生成的示教（确定性的）上，去掉 CVAE 几乎没有影响；在人类示教上，两个仿真任务的平均成功率从 35.3% 掉到 2%。训练时，编码器把示教中“观测解释不了的差异”交给 \(z\)，解码器不必把这些差异平均进同一个输出里，学到的映射更干净。测试时取 \(z=0\)，大致对应“最典型的风格”。在真实任务里，四路相机的丰富观测往往已经决定了该怎么做，真正 50:50 的分叉比我们的玩具任务少得多。

The encoder has indeed learned to put left and right demonstrations on opposite sides of \(z\), and the decoder indeed switches sides when \(z\) changes sign. But \(z=0\) lies near the boundary between them: over the stretch of \(o\) where both sides are about equally likely it decodes to "between the peaks" (the exact boundary drifts during training, so it is not necessarily at \(o=0\)), and that is ACT's failing stretch in FIG 2 (Pitfall 5 of Lesson 19.1 made the same point). Why does ACT use a CVAE at all, then? The paper's ablation answers: on scripted (deterministic) demonstrations, removing the CVAE made almost no difference; on human demonstrations, the average success over two simulated tasks fell from 35.3% to 2%. During training the encoder hands the variation that the observation cannot explain to \(z\), so the decoder need not average that variation into one output and learns a cleaner mapping. At test time \(z=0\) corresponds roughly to "the most typical style". In real tasks the rich observation from four cameras often already determines what to do, and true 50:50 forks are much rarer than in our toy task.

Diffusion Policy 把 19.6 节的 DDPM 原样搬过来，只改了两处：被生成的不再是图像，而是一整段动作 \(\mathbf A_t\)；去噪网络多了一个条件输入，即观测 \(\mathbf O_t\)。训练时取一段示教动作 \(\mathbf A^0_t\)，随机选一个噪声级别 \(k\)（19.6 节记作 \(t\)；这里 \(t\) 留给机器人的时间步，所以沿用 Diffusion Policy 论文的写法，用 \(k\)），加噪成 \(\mathbf A^k_t=\sqrt{\bar\alpha_k}\mathbf A^0_t+\sqrt{1-\bar\alpha_k}\,\boldsymbol\epsilon\)，让网络从 \((\mathbf O_t,\mathbf A^k_t,k)\) 预测出加进去的噪声 \(\boldsymbol\epsilon\)。生成时从纯噪声 \(\mathbf A^K_t\) 出发，一步步减去预测的噪声。观测只作为条件，不被生成，这让推理快得多。

多模态在这里是“免费”的：去噪是一个从噪声出发的随机过程，不同的初始噪声被推向不同的峰。下图展示 FIG 2 训练的扩散网络在决策点（第 8 步查询时的状态，夹爪在中线上）怎样把 30 个随机初始动作块一步步推成左右两类。“每块都选定一边”这件事，19.6 节 FIG 6 已经用精确的 score 展示过，19.7 节 FIG 5 也用现场训练的 flow matching 头展示过；这里的看点是 FIG 2 里那个只训练了几千步的网络还剩下什么毛病：少数落在两峰之间的样本，以及去噪步数对它们影响多大。

Diffusion Policy takes the DDPM of Lesson 19.6 as it is and changes two things: what is generated is no longer an image but a whole action sequence \(\mathbf A_t\), and the denoising network gets an extra conditioning input, the observation \(\mathbf O_t\). In training, take a demonstrated action sequence \(\mathbf A^0_t\), pick a noise level \(k\) at random (Lesson 19.6 calls it \(t\); here \(t\) is the robot's time step, so we follow the Diffusion Policy paper and write \(k\)), noise it into \(\mathbf A^k_t=\sqrt{\bar\alpha_k}\mathbf A^0_t+\sqrt{1-\bar\alpha_k}\,\boldsymbol\epsilon\), and have the network predict the added noise \(\boldsymbol\epsilon\) from \((\mathbf O_t,\mathbf A^k_t,k)\). To generate, start from pure noise \(\mathbf A^K_t\) and subtract the predicted noise step by step. The observation is only a condition and is never generated, which makes inference much faster.

Multimodality comes "for free" here: denoising is a random process starting from noise, and different initial noises are pushed to different peaks. The figure below shows how the diffusion network trained in FIG 2 pushes 30 random initial chunks step by step into a left group and a right group at the decision point (the state at the query of step 8, gripper on the centre line). That every chunk commits to one side was already shown by FIG 6 of Lesson 19.6 with the exact score, and by FIG 5 of Lesson 19.7 with a flow-matching head trained live; what to look for here is what is still wrong with FIG 2's network after a few thousand steps: the few samples that land between the peaks, and how much the number of denoising steps matters for them.

开始时样本是随机散开的噪声；随着去噪推进，它们聚成两支，各自落到一个峰上。\(|o|\) 小时左右大致各半。DDIM（Song 等，ICLR 2021）让训练用 \(K\) 步、推理只走其中的一部分：Diffusion Policy 在真实机器人上用 100 步训练、10 步推理，在 RTX 3080 上一次推理约 0.1 秒。去噪步数是计算量和质量之间的旋钮，每一步都是一次完整的网络前向。在我们这个一维动作块的小任务里，从 2 步到 50 步，落在两峰之间的样本比例只有若干个百分点（不到十个）的起伏，并不随步数单调变好；用默认的 10 步，训练三千到六千步后，决策点上 100 个样本里仍有大约 10 到 16 个撞上，它们来自网络本身还不够准，而不是步数太少。这些就是 FIG 2 里扩散策略那些零星的失败。

论文比较了两种去噪网络。CNN 版采用 Janner 等（Diffuser）的一维时间卷积 U-Net，观测特征和去噪步 \(k\) 通过 FiLM（逐通道的仿射调制）注入每个卷积层；作者建议新任务先用它，它几乎不需要调参。Transformer 版（时间序列扩散 Transformer）把带噪的动作当作 token，去噪步的正弦编码作为第一个 token，观测嵌入通过交叉注意力进入每个解码器块，动作 token 之间用因果掩码；在动作变化快、变化剧烈（如速度控制）的任务上更好，但对超参数更敏感（17.3 节）。视觉编码器是不预训练的 ResNet-18，把全局平均池化换成空间 softmax，把 BatchNorm 换成 GroupNorm（为了和 EMA 配合稳定）。

At the start the samples are random noise spread all over; as denoising proceeds they gather into two branches, each landing on a peak. For small \(|o|\) left and right are roughly half and half. DDIM (Song et al., ICLR 2021) lets training use \(K\) steps while inference visits only some of them: on real robots, Diffusion Policy trains with 100 steps and infers with 10, about 0.1 seconds per inference on an RTX 3080. The number of denoising steps is a knob between compute and quality, each step being a full forward pass of the network. In our small task with one-dimensional chunks, going from 2 to 50 steps moves the share of between-peak samples by several percentage points at most (under ten), and not steadily for the better; with the default 10 steps, after three to six thousand training steps, roughly 10 to 16 of 100 samples at the decision point still collide, and they come from the network not yet being accurate enough rather than from too few steps. These are the scattered failures of the diffusion policy in FIG 2.

The paper compares two denoising networks. The CNN version adopts the 1D temporal-convolution U-Net of Janner et al. (Diffuser), with the observation features and the denoising step \(k\) injected into every convolution layer by FiLM (channel-wise affine modulation); the authors recommend starting with it on a new task, as it needs almost no tuning. The transformer version (the time-series diffusion transformer) treats the noisy actions as tokens, puts a sinusoidal embedding of the denoising step first, feeds the observation embedding into every decoder block by cross-attention, and uses a causal mask among the action tokens; it does better on tasks whose actions change quickly and sharply (such as velocity control) but is more sensitive to hyperparameters (Lesson 17.3). The visual encoder is a ResNet-18 without pretraining, with global average pooling replaced by a spatial softmax and BatchNorm replaced by GroupNorm (for stable training together with the EMA).

Diffusion Policy 用三个时域描述执行方式：观测时域 \(T_o\)（看最近几帧观测）、预测时域 \(T_p\)（一次生成多少步）、执行时域 \(T_a\)（执行其中多少步再重新规划）。这就是 13.4 节 MPC 的滚动时域思想，只不过“优化器”换成了一次条件采样。\(T_a\) 越小越灵敏；可对一个会采样的策略来说，\(T_a\) 太小意味着相邻两次采样可能选了不同的峰，机器人在两种绕法之间来回摇摆。两个极端前面都见过：22.1 节 FIG 5 里每一步独立采样的分类头会来回抖，19.4 节 FIG 6 里一个块内各步独立采样会左右乱跳。\(T_a\) 就是介于“每步重采样”和“整块坚持到底”之间的旋钮，这里用一个真正训练出来的扩散策略来拧它。我们的任务里 \(T_p=8\)，下图比较不同的 \(T_a\)。

Diffusion Policy describes execution with three horizons: the observation horizon \(T_o\) (how many recent observations it sees), the prediction horizon \(T_p\) (how many steps one sample generates) and the execution horizon \(T_a\) (how many of them are executed before replanning). This is the receding-horizon idea of MPC in Lesson 13.4, with the "optimizer" replaced by one conditional sample. A smaller \(T_a\) is more responsive; but for a policy that samples, too small a \(T_a\) means two consecutive samples may pick different peaks, and the robot wavers between the two ways around. Both extremes have appeared before: in FIG 5 of Lesson 22.1 a classification head sampling every step independently dithers, and in FIG 6 of Lesson 19.4 the steps of one chunk sampled independently zigzag. \(T_a\) is the knob between "resample every step" and "keep the whole chunk", turned here on a diffusion policy that was actually trained. In our task \(T_p=8\); the figure below compares values of \(T_a\).

\(T_a=1\) 时每一步都重新采样，在决策区里第 \(t\) 步选左、第 \(t+1\) 步选右的情况并不少见，轨迹会犹豫着冲向中间。\(T_a=8\) 时一个样本的选择至少坚持 8 步，足够把夹爪带离分叉点，之后观测本身就说明了“已经在走左边”，后面的样本自然跟上。论文发现 \(T_a=8\) 对他们测试的大多数任务最好（通常配 \(T_o=2\)、\(T_p=16\)），并指出执行时域太长会因为反应慢而掉成功率；分块执行加上位置控制，还让策略能容忍最多约 4 步的延迟而性能不降。反应速度的那一面，FIG 3 的“分块执行”已经展示过：块越长，意外发生后反应越晚。

With \(T_a=1\) every step samples anew, and in the decision region it is not rare that step \(t\) picks left and step \(t+1\) picks right, so the path hesitates towards the middle. With \(T_a=8\) one sample's choice is kept for at least 8 steps, enough to carry the gripper away from the fork; from then on the observation itself says "already going left", and later samples follow. The paper found \(T_a=8\) best for most tasks it tested (usually with \(T_o=2\) and \(T_p=16\)) and notes that too long an execution horizon loses success through slow reactions; chunked execution together with position control also let the policy tolerate up to about 4 steps of latency without losing performance. The responsiveness side was shown in FIG 3's "Execute chunks": the longer the chunk, the later the reaction to a surprise.

对一个固定的观测 \(\mathbf o\)，均方误差 \(\mathbb E\|\mathbf a-\hat{\mathbf a}\|^2\) 在 \(\hat{\mathbf a}=\mathbb E[\mathbf a\mid\mathbf o]\) 处最小，L1 误差 \(\mathbb E|a-\hat a|\) 在中位数处最小（逐维）。对 FIG 1 的两峰混合，均值相对障碍物的偏移是

For a fixed observation \(\mathbf o\), the squared error \(\mathbb E\|\mathbf a-\hat{\mathbf a}\|^2\) is minimized at \(\hat{\mathbf a}=\mathbb E[\mathbf a\mid\mathbf o]\), and the L1 error \(\mathbb E|a-\hat a|\) at the median (per dimension). For the two-peak mixture of FIG 1, the mean's offset from the obstacle is

它小于碰撞距离 0.27 当且仅当 \(|o|<\tfrac12\operatorname{artanh}(0.27/0.55)\approx0.27\)。中位数则只看哪边超过一半：\(p_L>\tfrac12\) 时它落在左峰上，\(p_L<\tfrac12\) 时在右峰上，\(p_L=\tfrac12\) 时两峰之间的任何值都是中位数。这个跳变就是光滑的网络在 FIG 2 里学不好的地方。

It is smaller than the collision distance 0.27 exactly when \(|o|<\tfrac12\operatorname{artanh}(0.27/0.55)\approx0.27\). The median only asks which side holds more than half: for \(p_L>\tfrac12\) it lies on the left peak, for \(p_L<\tfrac12\) on the right one, and at \(p_L=\tfrac12\) any value between the peaks is a median. That jump is what a smooth network learns poorly in FIG 2.

ACT 的策略是一个分块的条件生成模型 \(\pi_\theta(\mathbf a_{t:t+k}\mid\mathbf o_t,z)\)，与编码器 \(q_\phi(z\mid\mathbf a_{t:t+k},\bar{\mathbf o}_t)\) 一起最小化（\(\bar{\mathbf o}_t\) 是不含图像的观测，即关节位置）

ACT's policy is a chunked conditional generative model \(\pi_\theta(\mathbf a_{t:t+k}\mid\mathbf o_t,z)\), trained together with an encoder \(q_\phi(z\mid\mathbf a_{t:t+k},\bar{\mathbf o}_t)\) (where \(\bar{\mathbf o}_t\) is the observation without images, the joint positions) to minimize

这就是 19.1 节的 β-ELBO，只是把高斯似然换成了拉普拉斯似然（L1）。测试时 \(z=\mathbf 0\)。在第 \(t\) 步，记 \(\hat{\mathbf a}^{(s)}_t\) 为第 \(s\) 步查询对第 \(t\) 步的预测，\(s=t-k+1,\dots,t\)，按查询时间从早到晚编号为 \(i=0,1,\dots\)，执行的动作是

This is the β-ELBO of Lesson 19.1 with the Gaussian likelihood replaced by a Laplace one (L1). At test time \(z=\mathbf 0\). At step \(t\), let \(\hat{\mathbf a}^{(s)}_t\) be the prediction for step \(t\) made by the query at step \(s\), \(s=t-k+1,\dots,t\), numbered \(i=0,1,\dots\) from the earliest query to the latest; the executed action is

如果各次查询的误差相互独立、方差都是 \(\sigma^2\)，等权平均（\(m=0\)）把方差降到 \(\sigma^2/k\)，这就是 FIG 3 里抖动下降的原因。代价是第 \(t\) 步的动作里混着 \(k-1\) 步之前、还没看到最新情况的旧预测，所以反应时间和 \(k\) 同量级。

If the errors of different queries are independent with variance \(\sigma^2\), equal weights (\(m=0\)) reduce the variance to \(\sigma^2/k\), which is why the jitter drops in FIG 3. The price is that the action at step \(t\) contains old predictions made up to \(k-1\) steps earlier, before the latest events were seen, so the reaction time is of the order of \(k\).

取噪声表 \(\bar\alpha_k\)（我们用 Nichol 与 Dhariwal 的余弦表，论文也发现它最好），训练目标是 19.6 节的噪声预测损失，只多了条件 \(\mathbf O_t\)：

With a noise schedule \(\bar\alpha_k\) (we use the cosine schedule of Nichol & Dhariwal, which the paper also found best), the training objective is the noise-prediction loss of Lesson 19.6 with the extra condition \(\mathbf O_t\):

DDIM（\(\eta=0\)）从 \(\mathbf A^K\sim\mathcal N(\mathbf 0,I)\) 出发，在一串递减的步 \(k>k'\) 上迭代：先估计干净的动作块，再按新的噪声水平重新组合：

DDIM (\(\eta=0\)) starts from \(\mathbf A^K\sim\mathcal N(\mathbf 0,I)\) and iterates over a decreasing sequence of steps \(k>k'\): estimate the clean chunk, then recombine at the new noise level:

一次推理需要 \(n\) 次网络前向（\(n\) 是去噪步数），而 ACT 只需要一次。执行时只用生成的 \(T_p\) 步中的前 \(T_a\) 步（\(\hl{ta}{T_a\le T_p}\)），然后用新的观测重新采样。

One inference costs \(n\) forward passes of the network (\(n\) the number of denoising steps), while ACT needs one. Execution uses only the first \(T_a\) of the \(T_p\) generated steps (\(\hl{ta}{T_a\le T_p}\)) and then samples again from the new observation.

19.7 节的 flow matching 可以直接替换扩散：取 \(\mathbf A^{(0)}\sim\mathcal N(\mathbf 0,I)\) 和示教动作块 \(\mathbf A^{(1)}\)，沿直线插值 \(\mathbf A^{(\tau)}=(1-\tau)\mathbf A^{(0)}+\tau\mathbf A^{(1)}\)，训练一个速度场去回归直线的速度。这里的流时间写作 \(\tau\)（19.7 节写作 \(t\)，这里 \(t\) 留给机器人的时间步），方向与 19.7 节和 π0 相同：\(\tau=0\) 是噪声，\(\tau=1\) 是数据，正好和上面扩散的 \(k\)（\(k=0\) 是数据）相反。

The flow matching of Lesson 19.7 can replace diffusion directly: take \(\mathbf A^{(0)}\sim\mathcal N(\mathbf 0,I)\) and a demonstrated chunk \(\mathbf A^{(1)}\), interpolate along the straight line \(\mathbf A^{(\tau)}=(1-\tau)\mathbf A^{(0)}+\tau\mathbf A^{(1)}\), and train a velocity field to regress the line's velocity. The flow time is written \(\tau\) here (Lesson 19.7 writes \(t\), which here is the robot's time step), with the direction of Lesson 19.7 and π0: \(\tau=0\) is noise and \(\tau=1\) is data, the opposite of the diffusion index \(k\) above (\(k=0\) is data).

它和扩散策略的结构完全相同：条件生成一整段动作、多步积分、滚动执行；区别是训练目标更简单、采样路径更直，少量积分步就够。π0（Physical Intelligence，Black 等，2024）在一个视觉-语言模型上接了一个约 3 亿参数、用 flow matching 训练的“动作专家”，一次生成 \(H=50\) 步的动作块，推理时做 10 步欧拉积分，能以最高 50 Hz 控制机器人；训练时 \(\tau\) 从偏向噪声一端的 Beta 分布中采样（细节见 19.7 节）。

Its structure is exactly that of a diffusion policy: conditionally generate a whole chunk, integrate over several steps, execute in a receding horizon; the differences are a simpler training target and straighter sampling paths, so a few integration steps suffice. π0 (Physical Intelligence; Black et al., 2024) attaches to a vision-language model an "action expert" of about 300 million parameters, trained with flow matching, that generates action chunks of \(H=50\) steps with 10 Euler integration steps at inference and controls robots at up to 50 Hz; in training \(\tau\) is drawn from a Beta distribution that leans towards the noisy end (details in Lesson 19.7).

工程上的几条经验。动作用绝对的目标关节位置或末端位姿（位置控制），而不是速度或增量：两篇论文都报告了这一点（ACT 用增量关节位置时性能下降；Diffusion Policy 的位置控制明显好于速度控制）。归一化：Diffusion Policy 把每一维动作按数据中的最小值和最大值缩放到 \([-1,1]\)；一个异常样本就能把范围撑大（17.5 节误区 1），部署时还必须用同一套统计量还原。推理延迟：ACT 一次前向约 10 ms，可以 50 Hz 每步查询；Diffusion Policy 一次推理要多步去噪，常见做法是在机器人执行当前块的同时计算下一块，并用 DDIM 或蒸馏（一致性模型等）减少步数。数据量：两者在单任务上都只用几十到几百条示教，靠的是高质量的遥操作数据（22.3 节）。

Some engineering lessons. Use absolute target joint positions or end-effector poses as actions (position control), not velocities or increments: both papers report this (ACT degraded with delta joint positions; Diffusion Policy's position control clearly beat velocity control). Normalization: Diffusion Policy scales every action dimension to \([-1,1]\) by its minimum and maximum in the data; a single outlier stretches that range (Pitfall 1 of Lesson 17.5), and deployment must undo the scaling with the same statistics. Inference latency: one forward pass of ACT takes about 10 ms, so it can be queried at every step at 50 Hz; one inference of Diffusion Policy needs several denoising steps, and a common practice is to compute the next chunk while the robot executes the current one, and to reduce the steps with DDIM or distillation (consistency models and the like). Data: both use only tens to hundreds of demonstrations per task, relying on high-quality teleoperation data (Lesson 22.3).