模仿学习：BC 与 DAggerImitation Learning: BC & DAgger

叠衣服、插充电线、把杯子放进洗碗机：这些任务很难写成代价函数，更难写成控制律，但一个人用遥操作手柄做一遍却很容易（遥操作设备与数据采集见 22.3 节）。模仿学习就是从这样的示教里学出一个策略 \(\pi_\theta(o)\)：输入机器人此刻的观测 \(o\)（相机图像、关节角），输出动作 \(a\)。

最简单的做法叫行为克隆（behaviour cloning, BC）：把示教拆成一对对 \((o_t,a_t)\)，当作普通的监督学习数据集，用 16.1 节的回归（连续动作）或分类（离散动作）去拟合。今天大多数操作策略，从 ALOHA 上的 ACT、Diffusion Policy（22.2 节）到用大量示教训练的 RT-1 和各种 VLA，训练目标的核心都是行为克隆，只是网络和动作表示更讲究。

可监督学习有一个默认前提：测试数据和训练数据来自同一个分布（16.2 节）。策略一上机就把它打破了：机器人自己的动作决定了它下一步看到什么。它犯一个小错，就到了专家没去过的状态；在那里它没有数据，更容易犯错；再错就离得更远。这一节要回答三个问题：

• 这种误差累积有多严重？（答案：最坏情况下额外代价随任务长度按 \(T^2\) 增长，FIG 1、FIG 2。）
• 怎么修？（让专家在学习者自己到过的状态上打标签：DAgger 及其人类门控版本，FIG 3、FIG 4；或者示教时故意加噪声，FIG 1。）
• 示教里同一个状态有两种做法时怎么办？（均方误差会取平均，FIG 5；生成式策略见 22.2 节。）

Folding a shirt, plugging in a charging cable, loading a cup into a dishwasher: such tasks are hard to write as a cost function and harder still as a control law, yet easy for a person to do once with a teleoperation device (teleoperation hardware and data collection are in Lesson 22.3). Imitation learning learns a policy \(\pi_\theta(o)\) from such demonstrations: it takes the robot's current observation \(o\) (camera images, joint angles) and outputs an action \(a\).

The simplest version is behaviour cloning (BC): cut the demonstrations into pairs \((o_t,a_t)\), treat them as an ordinary supervised dataset, and fit them with the regression (continuous actions) or classification (discrete actions) of Lesson 16.1. Most of today's manipulation policies, from ACT on ALOHA and Diffusion Policy (Lesson 22.2) to RT-1 and the various VLAs trained on large demonstration sets, have behaviour cloning at the core of their training objective; they only use fancier networks and action representations.

But supervised learning assumes that test data come from the same distribution as the training data (Lesson 16.2). A policy running on the robot breaks that at once: the robot's own actions decide what it sees next. One small mistake takes it to a state the expert never visited; there it has no data and is more likely to err again; the next mistake takes it further still. This lesson answers three questions:

• How bad is this compounding? (Answer: in the worst case the extra cost grows like \(T^2\) in the task length, FIG 1 and FIG 2.)
• How do we fix it? (Have the expert label the states the learner itself reaches: DAgger and its human-gated version, FIG 3 and FIG 4; or inject noise while demonstrating, FIG 1.)
• What if the demonstrations show two ways to act in the same state? (Squared error averages them, FIG 5; generative policies are in Lesson 22.2.)

想象一个学开车的人，只看过教练开车的录像。教练开得很稳，车一直在车道正中，所以录像里从来没有“车已经偏向路边，该怎么回来”这种画面。学员自己上路，方向稍微打多一点，车偏了 20 cm。这个画面他没见过，只能猜；猜错了就偏得更多，更陌生，直到压线。

关键的区别在于数据是谁产生的。普通监督学习里，测试样本由世界给出，和模型无关。模仿学习里，测试时的状态由学习者自己的策略产生：设专家策略为 \(\pi^*\)，它跑出来的状态分布是 \(\hl{dstar}{d_{\pi^*}}\)，训练数据就来自这里；学到的策略 \(\hat\pi\) 跑出来的状态分布是 \(\hl{dpi}{d_{\hat\pi}}\)，它在部署时面对的是这个分布。\(\hat\pi\) 和 \(\pi^*\) 差一点点，\(d_{\hat\pi}\) 就可能和 \(d_{\pi^*}\) 差很多，因为差异会沿着时间累积。这就是 16.2 节的协变量偏移，只不过偏移是策略自己造成的；19.4 节的自回归模型在生成时喂入自己的输出，遇到的“暴露偏差”是同一件事。

下面所有图都在同一个场景里：一台仓库里的移动机器人（琥珀色）以 1 m/s 沿一条 0.9 m 宽、22.7 m 长的弯曲通道从左下走到右上，两边是货架，碰到就算失败。专家 \(\pi^*\) 是一个简单的控制器：沿通道方向走，同时以增益 3 /m 把自己拉回中线。学习者看到的不是坐标本身，而是 26 个数：\(x,y\) 和它们 12 种随机组合的正弦与余弦（随机傅里叶特征），代替相机那样的“丰富”传感器。像图像一样，机器人挪动一点，这些数就变很多，所以离示教路线 20 cm 的状态，在网络看来是完全陌生的输入。网络是 26 → 32 → 32 → 2 的 tanh MLP，输出前进方向，用均方误差训练。

Picture someone learning to drive only from videos of an instructor. The instructor drives smoothly and stays in the middle of the lane, so the videos never show "the car has drifted toward the kerb, here is how to get back". The learner takes the wheel, steers a little too much, and the car is 20 cm off. That view was never in the videos, so the learner guesses; a wrong guess puts the car further off and the view gets stranger, until it crosses the line.

The key difference is who generates the data. In ordinary supervised learning the test samples come from the world, regardless of the model. In imitation learning the test-time states are produced by the learner's own policy. Call the expert policy \(\pi^*\): the states it visits follow the distribution \(\hl{dstar}{d_{\pi^*}}\), and the training data come from there. The learned policy \(\hat\pi\) visits states distributed as \(\hl{dpi}{d_{\hat\pi}}\), and that is what it faces at deployment. A small difference between \(\hat\pi\) and \(\pi^*\) can make \(d_{\hat\pi}\) very different from \(d_{\pi^*}\), because the differences accumulate over time. This is the covariate shift of Lesson 16.2, except that the policy causes the shift itself; the "exposure bias" of the autoregressive models in Lesson 19.4, which feed their own outputs back in while generating, is the same thing.

Every figure below uses one scene: a mobile robot (amber) in a warehouse drives at 1 m/s along a winding corridor 0.9 m wide and 22.7 m long, from bottom left to top right, with shelves on both sides; touching a shelf is a failure. The expert \(\pi^*\) is a simple controller: it heads along the corridor and pulls itself back to the centre line with a gain of 3 /m. The learner does not see its coordinates directly but 26 numbers: \(x,y\) and the sines and cosines of 12 random combinations of them (random Fourier features), a stand-in for a "rich" sensor such as a camera. As with an image, a small move changes these numbers a lot, so a state 20 cm from the demonstrated route is an unfamiliar input to the network. The network is a 26 → 32 → 32 → 2 tanh MLP that outputs a heading direction and is trained with squared error.

按“训练”：网络在一条示教（约 450 个 \((o,a)\) 对）上做 3000 步 Adam（批大小 32，学习率 0.003），然后自动做 6 次评测。评测时机器人的轮子会打滑：执行的方向带一点有色噪声（标准差 0.2 rad），每次评测的噪声不同。专家在这 6 次评测里次次都到达终点。训练完后红色的“误差图”出现：每个格子的红色越深，学到的方向和专家方向的夹角越大。拖动琥珀色的探针圆圈看任意一点的两个方向。

Press "Train": the network takes 3000 Adam steps (batch 32, learning rate 0.003) on one demonstration (about 450 \((o,a)\) pairs) and is then evaluated 6 times. During evaluation the wheels slip: the executed heading carries a little coloured noise (standard deviation 0.2 rad), different in every evaluation run. The expert reaches the goal in all 6 of these runs. After training the red "error map" appears: the more red a cell, the larger the angle between the learned direction and the expert's. Drag the amber probe circle to compare the two directions anywhere.

默认设置下（1 条示教、无噪声），训练损失约 \(2\times10^{-4}\)，在示教路线 5 cm 以内，学到的方向和专家只差约 4°；可 6 次评测全部撞墙，最远的走了路线的 29%，最近的只走了 8%。误差图解释了原因：离示教路线 20 cm 以上的格子，平均夹角约 45°。示教状态离中线平均只有 3 mm，评测状态平均 13 cm：机器人一旦被打滑推离那条细线，就进入了网络从没见过的输入区，在那里它的方向基本是乱的。

多给示教并没有用：4 条无噪声示教几乎重合（起点只差几厘米，专家马上把自己拉回中线），数据量翻了 4 倍，覆盖的状态却一样窄，6 次评测仍然 0 次成功（2 条示教时是 1 次）。真正有用的是示教里出现过“偏了又回来”的状态：示教噪声 0.2 rad 时 3 次成功，0.25 到 0.35 rad 时 4 次，0.4 rad 以上 5 次。

With the defaults (1 demonstration, no noise) the training loss is about \(2\times10^{-4}\), and within 5 cm of the demonstrated route the learned direction is only about 4° off the expert's; yet all 6 evaluation runs hit a shelf, the furthest after 29% of the route and the shortest after only 8%. The error map shows why: in cells more than 20 cm from the demonstrated route the average angle is about 45°. Demonstration states are on average only 3 mm from the centre line, evaluation states 13 cm: once wheel slip pushes the robot off that thin line, it is in a region of inputs the network has never seen, and there its directions are essentially arbitrary.

More demonstrations do not help: 4 noise-free demonstrations nearly coincide (their starts differ by a few centimetres and the expert immediately pulls back to the centre line), so the data grow fourfold while the states they cover stay just as narrow, and the 6 runs still succeed 0 times (1 time with 2 demonstrations). What helps is demonstrations that contain "drifted off and came back" states: with 0.2 rad of demo noise 3 runs succeed, with 0.25 to 0.35 rad 4 runs, and from 0.4 rad on 5 runs.

把 FIG 1 抽象成一个最简单的模型。任务有 \(T\) 步，每步在专家的状态上，学习者以概率 \(\varepsilon\) 犯错并离开专家的轨迹；离开之后的每一步记代价 1（“不在该在的地方”）。

• BC 学习者：只在专家的状态上训练过，离开之后不知道怎么回来，就一直在外面。到第 \(t\) 步时已经离开的概率是 \(1-(1-\varepsilon)^t\approx\varepsilon t\)，把它对 \(t\) 求和，总代价约为 \(\hl{bcT2}{\varepsilon T^2/2}\)：按 \(T^2\) 增长。直到 \(\varepsilon T\) 接近 1（几乎每次都会掉下去）才变平，那时代价已接近 \(T\)。
• DAgger 学习者：在自己离开后的状态上也有专家标签，所以每步有概率 \(1/u\) 回到轨迹上（平均 \(u\) 步回来）。它在外面的时间比例约为 \(\varepsilon/(\varepsilon+1/u)\)，总代价约 \(\hl{dagT}{u\varepsilon T}\)：按 \(T\) 增长。

下图上半部分是 BC 学习者的 24 次模拟（每行一次，横轴是时间，紫色格子表示那一步在轨迹外），下半部分是 DAgger 学习者（蓝色格子）。右边的对数坐标图是期望代价随 \(T\) 的变化（精确值，实线）和两个上界（虚线，第 8 节推导）：对数坐标里斜率 2 和斜率 1 的直线。

Boil FIG 1 down to the simplest possible model. The task has \(T\) steps; at each step on the expert's states the learner errs with probability \(\varepsilon\) and leaves the expert's trajectory; every step spent off it costs 1 ("not where it should be").

• The BC learner was trained only on the expert's states and, once off, does not know how to get back, so it stays off. The probability that it has left by step \(t\) is \(1-(1-\varepsilon)^t\approx\varepsilon t\); summing over \(t\) gives a total cost of about \(\hl{bcT2}{\varepsilon T^2/2}\): it grows like \(T^2\). It only flattens when \(\varepsilon T\) approaches 1 (it falls off almost every time), and by then the cost is close to \(T\).
• The DAgger learner also has expert labels on the states it reaches after leaving, so each step it returns with probability \(1/u\) (it takes \(u\) steps on average). The fraction of time it spends off is about \(\varepsilon/(\varepsilon+1/u)\), and the total cost about \(\hl{dagT}{u\varepsilon T}\): it grows like \(T\).

The top half of the figure is 24 simulated runs of the BC learner (one per row, time along the horizontal axis, purple cells = off the trajectory at that step), the bottom half the DAgger learner (blue cells). The log–log plot on the right shows the expected cost against \(T\) (exact, solid) and the two upper bounds (dashed, derived in Section 8): straight lines of slope 2 and slope 1 on log axes.

默认 \(\varepsilon=0.02\)、\(u=5\)、\(T=100\) 时，BC 平均有 57.5 步在外面，DAgger 只有 8.8 步。这个玩具模型把 BC 想得很悲观（一旦离开永不回来），但它抓住了要点：BC 的问题不是单步误差 \(\varepsilon\) 大，而是一次误差的后果会延续到任务结束。所以任务越长（机器人每秒做 10 到 50 次决策，一个 30 秒的任务就是几百到上千步），BC 越吃亏。

With the defaults \(\varepsilon=0.02\), \(u=5\), \(T=100\), BC spends 57.5 steps off on average and DAgger only 8.8. The toy model is pessimistic about BC (once off, never back), but it captures the point: BC's problem is not a large per-step error \(\varepsilon\), but that the consequences of one error last until the end of the task. So the longer the task (a robot makes 10 to 50 decisions a second, so a 30-second task is hundreds to over a thousand steps), the worse BC fares.

既然问题是“训练分布 \(d_{\pi^*}\) 不是测试分布 \(d_{\hat\pi}\)”，最直接的修法就是到 \(d_{\hat\pi}\) 上去采数据。Ross、Gordon 和 Bagnell（2011）提出的 DAgger（Dataset Aggregation，数据聚合）就是这么做的：

If the problem is that the training distribution \(d_{\pi^*}\) is not the test distribution \(d_{\hat\pi}\), the most direct fix is to collect data on \(d_{\hat\pi}\). That is exactly what DAgger (Dataset Aggregation) by Ross, Gordon and Bagnell (2011) does:

常用的选择是 \(\beta_1=1\)、之后 \(\beta_i=0\)：第一轮就是专家示教，之后完全由学习者开。下图从 FIG 1 默认的 BC 策略（0/6）出发。每按一次“一轮 DAgger”：学习者带着打滑噪声跑一遍（琥珀色线），撞墙或到终点就停；它到过的每个状态由专家打标签（蓝色点），并入数据集，在全部数据上再训练 800 步，然后用和 FIG 1 完全相同的 6 次评测打分。

A common choice is \(\beta_1=1\) and \(\beta_i=0\) afterwards: the first round is the expert demonstration, and from then on the learner drives alone. The figure starts from FIG 1's default BC policy (0/6). Each press of "One DAgger round" lets the learner drive once with wheel slip (amber line) until it hits a shelf or reaches the goal; the expert labels every state it visited (blue dots), these are aggregated into the dataset, the network trains 800 more steps on all of it, and it is scored with exactly the same 6 evaluation runs as in FIG 1.

默认设置（聚合、\(\beta=0\)）下，6 次评测的成功次数依次是 0、2、3、5、6：4 轮后全部成功，训练集从 451 个样本涨到 1519 个。前几轮采集都撞墙了（每轮只到达路线的一段），但每一轮都恰好在“上一个策略出错的地方”补上了标签。“只用最新一轮”每一轮都是 0/6：新策略学会了上次出错的那一小段，却忘了其余的路。这就是“聚合”二字的分量：DAgger 的理论保证（第 8 节）依赖于在所有历史数据上训练。采集时取 \(\beta=0.5\)，专家每隔几步帮一把，采集一遍就能走完整条路线，第一轮之后就是 6/6，代价是每轮请专家标注更多的状态（每轮约 460 个）。

With the defaults (aggregate, \(\beta=0\)) the successes are 0, 2, 3, 5, 6: after 4 rounds all runs succeed, and the training set has grown from 451 to 1519 samples. The early collection runs all hit a shelf (each covers only part of the route), but every round adds labels exactly "where the previous policy went wrong". With "newest round only" every round scores 0/6: the new policy learns the stretch where the last one failed and forgets the rest of the route. That is the weight of the word "aggregation": DAgger's guarantee (Section 8) relies on training on all past data. With \(\beta=0.5\) during collection the expert steps in every few steps, a single collection run covers the whole route, and it is 6/6 after the first round, at the price of more labels per round (about 460).

DAgger 要求专家在学习者开车时随叫随到。FIG 1 的“示教噪声”滑块是一种更省事的办法：示教时就故意把专家推离轨迹，执行加了噪声的动作，但记录的标签是专家想要的动作。这样 \(d_{\pi^*}\) 本身就变宽，覆盖了学习者可能偏到的地方。Laskey 等（2017）的 DART 把这个思路系统化，并按学习者的误差大小来选噪声。噪声太小覆盖不够，太大则示教本身变差甚至失败。它的局限是：噪声只能模拟“附近”的偏差，学习者真正会犯的错误由它自己决定，DAgger 直接到那里采样。19.4 节的 FIG 4 已经在自回归轨迹模型上看过同样的效果（扰动示教状态、标上专家的回正动作）；这里换成闭环策略和轮子打滑。早在 1990 年前后，ALVINN 就用合成“车偏了”的图像来教网络回正，NVIDIA 2016 年的端到端驾驶网络用左右两侧相机的图像配上修正后的方向盘角，是同一个思想。

DAgger needs the expert on call while the learner drives. FIG 1's "demo noise" slider is a cheaper alternative: push the expert off its trajectory while demonstrating, executing a noisy action but recording the action the expert intended. Then \(d_{\pi^*}\) itself widens and covers places the learner may drift to. DART (Laskey et al., 2017) made this systematic and chose the noise to match the learner's errors. Too little noise covers too little; too much makes the demonstrations themselves worse or even fail. Its limit: noise only imitates nearby deviations, whereas the mistakes a learner really makes are decided by the learner itself, and DAgger samples exactly there. FIG 4 of Lesson 19.4 showed the same effect for an autoregressive trajectory model (perturb the demonstrated states, label them with the expert's way back); here it is a closed-loop policy with wheel slip. Around 1990 ALVINN already taught its network to recover with synthesized "the car has drifted" images, and NVIDIA's 2016 end-to-end driving network paired images from left and right side cameras with corrected steering angles: the same idea.

FIG 3 的专家是一段代码，随时能回答“在这个状态你会怎么走”。人类专家没那么容易：看着一个正在乱开的机器人，凭空说出每一帧该打多少方向，没有手上的反馈，标签质量很差；而且让一个半成品策略一直开下去，真机可能撞坏东西。HG-DAgger（Kelly 等，2019）的折中是：学习者开，人在旁边看着，觉得要出事时接管；只有接管期间人亲手开出来的 \((o,a)\) 被记录下来，接管结束交还控制权。每跑完一次，在所有数据上重新训练。

下图里你就是那个人：按住“接管”按钮时由专家驾驶（代表你的手），琥珀色线段就是被记录的标签；松开就交还给学习者（紫色线段）。也可以把门控交给“自动”：离中线超过 0.25 m 就接管，回到 0.08 m 以内交还。

FIG 3's expert is a piece of code that can answer "what would you do in this state?" at any moment. A human expert cannot do that so easily: watching a robot drive badly and stating, frame by frame, how much to steer, without feeling the result in your hands, gives poor labels; and letting a half-trained policy keep driving can damage a real robot. HG-DAgger (Kelly et al., 2019) compromises: the learner drives, a person watches and takes over when trouble is coming; only the \((o,a)\) pairs the person produces while in control are recorded, and control is handed back afterwards. After each run the policy is retrained on all data.

In the figure you are that person: while you hold the "Take over" button the expert drives (standing in for your hands) and the amber segments are the recorded labels; release it and the learner drives again (purple segments). You can also hand the gate to "auto": take over beyond 0.25 m from the centre line, hand back within 0.08 m.

用“自动”门控时，第 1 次接管了 107 步，第 2 次 10 步，第 3 次学习者已经自己跑完全程；这时一共 117 个标签，评测 6/6（之后偶尔还会有一次短暂的接管，比如第 7 次接管了 12 步）。FIG 3 的 DAgger 为了 6/6 新增了 1068 个标签，因为它给学习者到过的每一个状态都打标签，包括那些学习者本来就做对了的。HG-DAgger 只在人认为危险的地方要标签，代价是要有人一直盯着，而且“什么时候接管”本身是人的判断。后续工作让这个判断自动化：SafeDAgger（Zhang 与 Cho，2017）训练一个网络预测学习者会不会出错，EnsembleDAgger（Menda 等，2019）和 ThriftyDAgger（Hoque 等，2021）用策略集成的分歧等不确定性信号（21.3 节）来决定何时请人帮忙。

With the "auto" gate, the first run takes over for 107 steps, the second for 10, and on the third run the learner already completes the route by itself: 117 labels in total at that point, and the evaluation scores 6/6 (a later run still needs a short takeover now and then; the seventh, for instance, takes over for 12 steps). FIG 3's DAgger added 1068 labels to reach 6/6, because it labels every state the learner visits, including those where the learner was already right. HG-DAgger asks for labels only where a person judges it dangerous, at the cost of someone watching all the time, and "when to take over" is itself a human judgement. Later work automates that judgement: SafeDAgger (Zhang and Cho, 2017) trains a network to predict whether the learner is about to err, and EnsembleDAgger (Menda et al., 2019) and ThriftyDAgger (Hoque et al., 2021) use uncertainty signals such as the disagreement of an ensemble of policies (Lesson 21.3) to decide when to ask for help.

DAgger 解决的是“去哪里采数据”。还有一个问题出在“怎么拟合数据”上。人示教时，面对同一个场景经常有好几种都对的做法：从障碍物左边绕或右边绕，先抓左边的杯子或右边的杯子。用均方误差（MSE）训练时，网络在每个观测上输出的是数据里动作的条件均值 \(\hl{mean}{\mathbb E[a\mid o]}\)（推导见第 8 节）。一半示教向左、一半向右，平均就是直走。

19.1、19.2 和 17.5 节看过“一个动作”的版本。这里看闭环的版本：机器人以 1 m/s 向前（上）走，正前方 3 m 处有一根半径 0.4 m 的柱子，它每一步选择横向速度 \(a\)。它的传感器只有一个朝前的测距仪，所以观测只有“走了多远” \(y\)，看不到自己的横向位置。20 条示教中，一部分从左边绕（横向偏移最大 0.9 m），其余从右边绕。比较三种策略头，都是 32–32 的 tanh MLP：

• MSE 回归 \(a=\pi_\theta(y)\)：就是前面用的 BC。
• 分类 + 每步采样：把横向速度分成 5 档（−1.2、−0.6、0、0.6、1.2 m/s），用交叉熵训练，执行时每一步从预测的分布里独立采样一档。它能表示“两个都可能”（17.5 节的动作 token 就是这个思路）。
• 给定“走哪边” \(a=\pi_\theta(y,z)\)：额外告诉网络每条示教走的是哪一边 \(z=\pm1\)，执行时每次评测开始时抽一次 \(z\)，之后一直不变。这里我们直接把 \(z\) 交给它，是为了看清“承诺一个模式”有多大作用；19.1 节的条件 VAE 不需要这个标签，它自己学出这样一个隐变量。

DAgger fixes "where the data come from". A second problem lies in "how the data are fitted". Demonstrators often have several equally correct ways to act in the same scene: pass the obstacle on the left or on the right, pick up the left cup or the right one first. Trained with mean squared error (MSE), the network outputs the conditional mean of the actions in the data, \(\hl{mean}{\mathbb E[a\mid o]}\), at every observation (derived in Section 8). Half the demonstrations go left, half go right: the average goes straight.

Lessons 19.1, 19.2 and 17.5 showed the single-action version. Here is the closed-loop version: the robot drives forward (up) at 1 m/s, a pillar of radius 0.4 m stands 3 m ahead, and at every step it chooses a sideways speed \(a\). Its only sensor is a forward-facing range finder, so it observes only how far it has come, \(y\), and not its own sideways position. Of 20 demonstrations some pass on the left (up to 0.9 m sideways) and the rest on the right. Three policy heads are compared, all 32–32 tanh MLPs:

• MSE regression \(a=\pi_\theta(y)\): the BC used so far.
• Classification + sampling every step: the sideways speed is split into 5 levels (−1.2, −0.6, 0, 0.6, 1.2 m/s), trained with cross-entropy, and at run time each step samples a level independently from the predicted distribution. It can represent "either is possible" (the action tokens of Lesson 17.5 follow this idea).
• Told which side, \(a=\pi_\theta(y,z)\): the network is additionally told which side each demonstration took, \(z=\pm1\); at run time \(z\) is drawn once at the start of each trial and then kept. We hand it \(z\) directly to see how much "committing to a mode" buys; the conditional VAE of Lesson 19.1 needs no such label and learns a latent variable like it by itself.

左绕比例为 \(p\) 时，条件均值让机器人在柱子处的横向偏移是 \((1-2p)\times0.9\) m，要大于柱子半径 0.4 m 才能通过，即 \(p>0.72\) 或 \(p<0.28\)。FIG 5 里 MSE 头在 \(p=0.7\) 时撞上、\(p=0.75\) 时通过，和公式一致。分类头每步采样在 \(p=0.5\) 时 20 次全撞：它知道“左右都行”，可每一步都重新掷硬币，左一步右一步，横向位置像随机游走一样在中间徘徊，哪边都没去成。给定 \(z\) 的头一次也不撞。教训是：能表示多个模式还不够，执行时还要在一段时间里坚持同一个模式。这正是动作分块（一次预测并执行一串动作）、隐变量和扩散策略在 22.2 节要解决的问题。

With a left share \(p\), the conditional mean puts the robot \((1-2p)\times0.9\) m sideways at the pillar, which must exceed the pillar's radius of 0.4 m to pass, i.e. \(p>0.72\) or \(p<0.28\). In FIG 5 the MSE head hits the pillar at \(p=0.7\) and passes at \(p=0.75\), matching the formula. The classification head with per-step sampling hits the pillar in all 20 trials at \(p=0.5\): it knows "either side is fine", but it tosses a new coin at every step, one step left and one step right, so its sideways position wanders around the middle like a random walk and it never commits to either side. The head told \(z\) never hits it. The lesson: representing several modes is not enough; at run time the policy must also stick to one mode for a while. That is exactly what action chunking (predicting and executing a sequence of actions at once), latent variables and diffusion policies address in Lesson 22.2.

和 6.12 节一样用“代价”的语言：有限时域 \(T\)，状态 \(s_t\)，动作 \(a_t\)，每步代价 \(c(s_t,a_t)\in[0,1]\)，策略 \(\pi\) 的总代价 \(J(\pi)=\mathbb E\big[\sum_{t=1}^T c(s_t,a_t)\big]\)。\(d^t_\pi\) 是执行 \(\pi\) 时第 \(t\) 步的状态分布，\(d_\pi=\tfrac1T\sum_t d^t_\pi\) 是整条轨迹上的平均状态分布。专家 \(\pi^*\) 是确定性的。学习者 \(\hat\pi\) 在状态 \(s\) 上的误差用 0–1 损失 \(\ell(s,\hat\pi)=\mathbb 1[\hat\pi(s)\ne\pi^*(s)]\) 衡量（连续动作时换成任何能上界它的代理损失）。

行为克隆是在专家分布上做经验风险最小化：

Use the language of cost, as in Lesson 6.12: a finite horizon \(T\), states \(s_t\), actions \(a_t\), a per-step cost \(c(s_t,a_t)\in[0,1]\), and the total cost of a policy \(\pi\), \(J(\pi)=\mathbb E\big[\sum_{t=1}^T c(s_t,a_t)\big]\). \(d^t_\pi\) is the distribution of the state at step \(t\) when running \(\pi\), and \(d_\pi=\tfrac1T\sum_t d^t_\pi\) the average state distribution along the trajectory. The expert \(\pi^*\) is deterministic. The learner's error at state \(s\) is the 0–1 loss \(\ell(s,\hat\pi)=\mathbb 1[\hat\pi(s)\ne\pi^*(s)]\) (for continuous actions, any surrogate loss that upper-bounds it).

Behaviour cloning is empirical risk minimization on the expert's distribution:

对随机策略 \(\pi_\theta(a\mid o)\)，这就是最大似然 \(\max_\theta\sum\log\pi_\theta(a_t\mid o_t)\)：固定方差的高斯策略给出 MSE，离散动作给出交叉熵（16.2 节的“损失 = 负对数似然”）。\(\varepsilon\) 是监督学习能控制的量：数据越多、模型越好，它越小，并且有泛化界保证它在专家分布上的值。

For a stochastic policy \(\pi_\theta(a\mid o)\) this is maximum likelihood, \(\max_\theta\sum\log\pi_\theta(a_t\mid o_t)\): a Gaussian policy with fixed variance gives MSE, discrete actions give cross-entropy ("losses are negative log-likelihoods", Lesson 16.2). \(\varepsilon\) is what supervised learning controls: more data and a better model make it smaller, and generalization bounds guarantee its value on the expert's distribution.

证明梗概（耦合论证）。让 \(\hat\pi\) 和 \(\pi^*\) 共用同一串随机数（同样的初始状态和转移噪声）。只要 \(\hat\pi\) 在前 \(k\) 步都和专家选了同一个动作，两条轨迹就完全一样，记 \(\tau\) 为 \(\hat\pi\) 第一次选了和专家不同动作的那一步，那么“\(\tau\le t\)”意味着“在专家轨迹的某个第 \(k\le t\) 步上 \(\hat\pi(s_k)\ne\pi^*(s_k)\)”。由并集界，

Proof sketch (a coupling argument). Let \(\hat\pi\) and \(\pi^*\) share one stream of randomness (the same initial state and transition noise). As long as \(\hat\pi\) picks the expert's action for the first \(k\) steps, the two trajectories are identical, so, writing \(\tau\) for the first step at which \(\hat\pi\) picks a different action from the expert, "\(\tau\le t\)" implies "\(\hat\pi(s_k)\ne\pi^*(s_k)\) at some step \(k\le t\) of the expert's trajectory". By the union bound,

分岔之前两条轨迹的代价相同，分岔之后每步最多多付 1，所以额外代价 \(\le\sum_{t=1}^T P(\tau\le t)\le T\cdot T\varepsilon\)。这个界是紧的：Ross 与 Bagnell 构造了让 BC 真的付出 \(\Theta(T^2\varepsilon)\) 的例子，FIG 2 的“掉下去就回不来”模型就是这一类（在 \(\varepsilon T\ll1\) 时代价 \(\approx\varepsilon T^2/2\)）。直觉上，\(T^2\) 里一个 \(T\) 来自“有 \(T\) 次机会犯第一个错”，另一个 \(T\) 来自“犯错之后还要付最多 \(T\) 步的代价”。

Before divergence the two trajectories pay the same cost; after it the learner pays at most 1 extra per step, so the extra cost is \(\le\sum_{t=1}^T P(\tau\le t)\le T\cdot T\varepsilon\). The bound is tight: Ross and Bagnell construct examples where BC really pays \(\Theta(T^2\varepsilon)\), and FIG 2's "once off, never back" model is of this kind (cost \(\approx\varepsilon T^2/2\) for \(\varepsilon T\ll1\)). Intuitively, one \(T\) in \(T^2\) comes from "\(T\) chances to make the first mistake", the other from "after the mistake, up to \(T\) more steps of cost".

记 \(Q^*_t(s,a)\) 为“第 \(t\) 步在 \(s\) 做 \(a\)，之后按专家走”的期望剩余代价（6.12 节的 cost-to-go，加上一步动作），\(V^*_t(s)=Q^*_t(s,\pi^*(s))\)。性能差分引理（Kakade 与 Langford，2002 的有限时域形式）把两个策略的代价差写成学习者自己的状态分布上的逐步“劣势”之和：

Let \(Q^*_t(s,a)\) be the expected remaining cost of doing \(a\) in \(s\) at step \(t\) and following the expert afterwards (the cost-to-go of Lesson 6.12 with one action prepended), and \(V^*_t(s)=Q^*_t(s,\pi^*(s))\). The performance difference lemma (in the finite-horizon form of Kakade and Langford, 2002) writes the cost difference of two policies as a sum of per-step "disadvantages" over the learner's own state distribution:

这里 \(u\) 是“走错一步之后，按专家继续走最多多付多少”：\(Q^*_t(s,a)-V^*_t(s)\le u\)。专家能很快纠正错误的任务（比如 FIG 1 的通道，专家 1 秒内就能拉回中线），\(u\) 是个小常数；FIG 2 的 \(u\) 就是平均回来需要的步数。关键在于 \(\hat\varepsilon\) 是在 \(d_{\hat\pi}\) 上量的：如果我们在学习者自己的分布上把监督误差压小，代价就只按 \(T\) 线性增长。

难点是 \(d_{\hat\pi}\) 取决于 \(\hat\pi\)，而 \(\hat\pi\) 又取决于训练数据，像追自己的尾巴。DAgger 把它看成在线学习：第 \(i\) 轮的损失函数是 \(\ell_i(\pi)=\mathbb E_{s\sim d_{\pi_i}}[\ell(s,\pi)]\)，在聚合数据 \(\hl{agg}{\mathcal D}\) 上训练就是在所有历史损失之和上取最优，即在线学习里的“跟随领先者”（Follow-The-Leader）。只要这个学习过程“无悔”（平均损失追得上事后最好的那个固定策略），Ross 等（2011）就证明：当轮数足够多（随 \(uT\) 增长，差一个对数因子），并且 \(\beta_i\) 衰减得足够快时，序列里至少有一个策略满足

Here \(u\) bounds how much extra cost one wrong step causes if the expert takes over afterwards: \(Q^*_t(s,a)-V^*_t(s)\le u\). In tasks where the expert corrects mistakes quickly (FIG 1's corridor, where the expert is back on the centre line within a second), \(u\) is a small constant; FIG 2's \(u\) is exactly the average number of steps to get back. The crux is that \(\hat\varepsilon\) is measured on \(d_{\hat\pi}\): if we make the supervised error small on the learner's own distribution, the cost grows only linearly in \(T\).

The difficulty is that \(d_{\hat\pi}\) depends on \(\hat\pi\), which depends on the training data: chasing one's own tail. DAgger treats it as online learning: round \(i\)'s loss is \(\ell_i(\pi)=\mathbb E_{s\sim d_{\pi_i}}[\ell(s,\pi)]\), and training on the aggregated data \(\hl{agg}{\mathcal D}\) minimizes the sum of all past losses, which is "Follow-The-Leader" in online learning. As long as this learner is "no-regret" (its average loss keeps up with the best fixed policy in hindsight), Ross et al. (2011) prove that with enough rounds (growing with \(uT\), up to a log factor) and with \(\beta_i\) decaying fast enough, at least one policy in the sequence satisfies

其中 \(\varepsilon_N\) 是事后看来最好的策略在所有轮的混合分布上的误差，也就是“如果我们事先知道学习者会去哪里，监督学习能做到多好”。实践中通常直接取最后一个策略，或者用验证集选。FIG 3 里“只用最新一轮”丢掉了聚合，相当于每轮只追最新的损失 \(\ell_i\)，这不是无悔的，结果来回摇摆。

噪声注入换一个角度看：执行 \(\pi^*\) 时加噪声，等于把训练分布从 \(d_{\pi^*}\) 换成 \(d_{\tilde\pi^*}\)（\(\tilde\pi^*\) 表示加了噪声的专家）。如果噪声造成的偏差和学习者误差造成的偏差差不多大，后者大部分落在前者覆盖的范围里，\(\hat\varepsilon\) 和 \(\varepsilon\) 就差不多；DART 正是按这个目标去选噪声的。

where \(\varepsilon_N\) is the error of the best policy in hindsight on the mixture of all rounds' distributions, i.e. "how well supervised learning could do if we knew in advance where the learner would go". In practice one usually takes the last policy or picks one on a validation set. FIG 3's "newest round only" drops the aggregation and chases only the latest loss \(\ell_i\) each round, which is not no-regret, and it oscillates.

Noise injection from this angle: executing \(\pi^*\) with noise replaces the training distribution \(d_{\pi^*}\) by \(d_{\tilde\pi^*}\) (\(\tilde\pi^*\) denotes the noisy expert). If the deviations caused by the noise are about as large as those caused by the learner's errors, most of the latter fall inside the region the former covers, and \(\hat\varepsilon\) is close to \(\varepsilon\); DART chooses its noise with exactly that goal.

对固定的观测 \(o\)，最小化 \(\mathbb E[(f-a)^2\mid o]=f^2-2f\,\mathbb E[a\mid o]+\mathbb E[a^2\mid o]\)，对 \(f\) 求导置零：

For a fixed observation \(o\), minimize \(\mathbb E[(f-a)^2\mid o]=f^2-2f\,\mathbb E[a\mid o]+\mathbb E[a^2\mid o]\); setting the derivative in \(f\) to zero gives

\(a_L=-a_R\) 时 \(f^*=(1-2p)a_R\)，FIG 5 的 \((1-2p)\times0.9\) m 就是把它沿路径积分的结果。L1 损失给出条件中位数，\(p=0.5\) 时区间 \([a_L,a_R]\) 里任何一点都是最优解，同样可能落在中间。根子在于：MSE 是固定方差高斯的负对数似然，而单峰高斯只能用“放在中间、方差变大”来解释两个峰。解决办法是换一个能表示多峰的 \(p_\theta(a\mid o)\)：混合高斯（混合密度网络）、离散化后的分类（17.5 节）、隐变量模型（19.1 节）、能量模型（19.5 节的隐式 BC）或扩散与 flow matching（22.2 节）。FIG 5 还说明了第二点：即便 \(p_\theta\) 是对的，逐步独立采样也会在模式间来回跳，所以这些方法还要在时间上保持一致（动作分块、固定隐变量）。

With \(a_L=-a_R\), \(f^*=(1-2p)a_R\); FIG 5's \((1-2p)\times0.9\) m is this integrated along the path. The L1 loss gives the conditional median, and at \(p=0.5\) every point of \([a_L,a_R]\) is optimal, so it too may land in the middle. The root cause: MSE is the negative log-likelihood of a fixed-variance Gaussian, and a unimodal Gaussian can only explain two peaks by "sitting in the middle with a wide spread". The cure is a \(p_\theta(a\mid o)\) that can represent several peaks: a Gaussian mixture (mixture density network), classification over discretized actions (Lesson 17.5), latent-variable models (Lesson 19.1), energy-based models (the implicit BC of Lesson 19.5) or diffusion and flow matching (Lesson 22.2). FIG 5 makes a second point too: even with the right \(p_\theta\), sampling independently at every step jumps between modes, so these methods must also stay consistent over time (action chunks, a fixed latent).

真正搭一个 BC 系统时，除了网络结构，下面这几个选择对成败的影响往往更大。

When you actually build a BC system, these choices often matter more than the network architecture.