Part V · 第 19 章生成模型 · 19.4Part V · Ch. 19 Generative Models · 19.4

自回归模型 (Autoregressive Models)Autoregressive Models

自回归模型把一个高维分布拆成一串一维的条件分布：先生成第一个元素，再在它的基础上生成第二个，依此类推。概率的链式法则保证这样的拆分是精确的，于是似然可以精确计算，训练就是每一步的交叉熵（teacher forcing），采样就是一步一步地抽。GPT 这样生成文字，PixelCNN 这样生成图像，RT-2、OpenVLA 这样的 VLA 也这样一个 token 一个 token 地输出机器人动作。本节在浏览器里现场训练小的自回归模型，看它怎样抓住多维动作之间的相关性，看温度、top-k 和 top-p 怎样在“多样”与“靠谱”之间取舍，再看它的代价：生成时喂的是自己的输出，训练时却从没见过自己的错误（exposure bias），误差会一步步累积。 An autoregressive model splits a high-dimensional distribution into a chain of one-dimensional conditionals: generate the first element, then the second given the first, and so on. The chain rule of probability makes this split exact, so the likelihood can be computed exactly, training is a cross-entropy at every step (teacher forcing), and sampling is drawing one step at a time. GPT generates text this way, PixelCNN generates images this way, and VLAs such as RT-2 and OpenVLA emit robot actions this way, one token at a time. This lesson trains small autoregressive models live in the browser, shows how they capture the correlations between action dimensions and how temperature, top-k and top-p trade diversity against reliability, and then shows the price: at generation time the model is fed its own outputs, although training never showed it its own mistakes (exposure bias), so errors accumulate step by step.

约 40 分钟~40 min 前置：4.2 条件概率 · 4.11 信息论 · 16.1 回归与分类 · 17.3 Transformer · 17.5 TokenizationPrereqs: 4.2 Conditional probability · 4.11 Information theory · 16.1 Regression & classification · 17.3 Transformers · 17.5 Tokenization 6 个交互6 interactives

1为什么机器人需要它Why robots need this

机器人策略要输出的往往不是一个数，而是一串有结构的东西：一个 7 维的末端动作（各维之间相互关联），未来一秒的一段动作（action chunk），一条轨迹，或者一句“先抓杯子再放进水槽”的计划。示教数据里，这些东西常常是多峰的：绕障碍物可以从左走，也可以从右走。16.2 节说过，用均方误差回归只能学到条件均值，两种走法一平均，正好撞上障碍。我们需要的是一个能表示整个分布、并且能从中采样的模型。

直接给一串长度为 $T$ 的序列建表是不可能的：每个元素有 $V$ 种取值，联合分布就有 $V^T$ 格。自回归模型用一个简单而精确的办法绕开它：一次只预测一个元素，条件是前面已经有的所有元素。每一步都只是一个普通的分类问题（16.1 节的 softmax 和交叉熵），而所有步的乘积恰好是联合分布。17.5 节把动作切成了 token，17.3 节给了能看全部历史的因果 Transformer；本节讲把它们串起来的那个概率模型：怎样训练、怎样采样，以及它最典型的毛病。

What a robot policy must output is often not one number but a sequence with structure: a 7-dimensional end-effector action (whose dimensions are correlated), the next second of actions (an action chunk), a trajectory, or a plan such as "pick up the cup, then put it in the sink". In demonstration data these are often multimodal: you can go around an obstacle on the left or on the right. Lesson 16.2 showed that regression with mean squared error only learns the conditional mean, and averaging the two ways around lands right on the obstacle. We need a model that represents the whole distribution and can sample from it.

Tabulating a sequence of length $T$ directly is hopeless: with $V$ values per element the joint distribution has $V^T$ cells. An autoregressive model avoids this in a simple and exact way: predict one element at a time, conditioned on all the elements already there. Every step is an ordinary classification problem (the softmax and cross-entropy of Lesson 16.1), and the product of all the steps is exactly the joint distribution. Lesson 17.5 cut actions into tokens and Lesson 17.3 provided the causal transformer that can look at the whole history; this lesson covers the probabilistic model that ties them together: how it is trained, how it is sampled, and its most typical failure.

2直觉：链式法则把一根线段切了又切Intuition: the chain rule slices a line segment again and again

一个移动机器人在三个路口依次选择：左转 ←、直行 ↑、右转 →。三步的选择是一个长度为 3 的序列 $\mathbf x=(x_1,x_2,x_3)$，共 27 种。4.2 节的乘法法则反复用两次，就得到链式法则：

A mobile robot makes three successive choices at three junctions: turn left ←, go straight ↑, turn right →. The three choices form a sequence of length 3, $\mathbf x=(x_1,x_2,x_3)$, with 27 possibilities. Applying the product rule of Lesson 4.2 twice gives the chain rule:

$$ \hl{joint}{p(x_1,x_2,x_3)}=p(x_1)\;p(x_2\mid x_1)\;p(x_3\mid x_1,x_2) $$

把它画出来：把长度为 1 的线段按 $p(x_1)$ 切成三段；每一段再按 $p(x_2\mid x_1)$ 切成三段；再按 $p(x_3\mid x_1,x_2)$ 切一次。最后一行的 27 小段，每段的宽度就是一个完整序列的概率，全部加起来正好是 1。采样就是在 $[0,1]$ 上随机扔一个点 $u$，看它落在哪一小段（4.9 节的逆变换采样）；等价地，也可以一步一步抽：先按第一行抽 $x_1$，再在那一段里按第二行抽 $x_2$……这叫祖先采样（ancestral sampling）。

下图的数字是故意设计的：第一步直行 ↑ 最常见（0.45），但直行之后去哪儿都差不多；左转 ← 不常见（0.25），可一旦左转，接下来几乎一定是 ←←，那是一个固定的掉头动作。

Draw it: cut a segment of length 1 into three pieces in proportion to $p(x_1)$; cut each piece into three by $p(x_2\mid x_1)$; cut once more by $p(x_3\mid x_1,x_2)$. Each of the 27 small pieces in the last row has a width equal to the probability of one complete sequence, and together they add up to exactly 1. Sampling is throwing a random point $u$ onto $[0,1]$ and seeing which small piece it lands in (the inverse-transform sampling of Lesson 4.9); equivalently, draw one step at a time: $x_1$ from the first row, then $x_2$ from the second row inside that piece, and so on. This is called ancestral sampling.

The numbers in the figure are chosen on purpose: going straight ↑ is the most common first move (0.45), but after going straight every continuation is about equally likely; turning left ← is less common (0.25), but once you turn left the next two moves are almost surely ←←, a fixed U-turn manoeuvre.

FIG 1链式法则：每一行在上一行的每一段里再切一次The chain rule: every row slices each piece of the row above

三行从上到下是三个因子，按所选的分解顺序排列。颜色表示这一步选了哪个动作：蓝 = ←，绿 = ↑，琥珀 = →。点击任意一段，选中这个前缀（选中路径上的段画成实色，其余变淡），读数给出链式法则的乘积；最后一行每段的宽度就是整条序列的概率。“采样”在底部的 $[0,1]$ 上扔一个均匀随机点 $u$（青色竖线），并选中它落入的序列。“贪心”每一步都选当前最可能的动作；“束搜索”每一步保留概率最大的 2 或 3 个前缀。 The three rows, top to bottom, are the three factors in the chosen factorization order. Colour tells which move a step chose: blue = ←, green = ↑, amber = →. Click any piece to select that prefix (pieces on the selected path are drawn solid, the rest fade); the readout gives the chain-rule product, and the width of each piece in the last row is the probability of the whole sequence. "Sample" throws a uniform random point $u$ onto $[0,1]$ at the bottom (the cyan vertical line) and selects the sequence it lands in. "Greedy" takes the most likely move at every step; "Beam" keeps the 2 or 3 most probable prefixes at every step.

试一试Try this

先按“贪心”：它选了 ↑←↑，概率只有 0.061。再自己点击，找出一条概率是它 3 倍以上的完整序列（最后一行的一段）。Press "Greedy" first: it chooses ↑←↑ with probability only 0.061. Then click your way to a complete sequence (a piece of the last row) that is more than 3 times as probable.
把分解顺序换成“x₃ 先”，再选中序列 ←←←：三行的切法全变了，但它的宽度（0.191）不变。Switch the factorization order to "x₃ first" and select the sequence ←←←: every row is sliced differently, yet its width (0.191) is unchanged.

核心洞见Key insight

链式法则对任何分布、任何顺序都精确成立，没有近似。自回归模型的全部工作，就是用一个网络去学每一个条件分布 $p(x_t\mid x_{<t})$。顺序不影响能否表示，但会影响每个条件分布好不好学，也影响贪心解码的结果：在“x₃ 先”的顺序下，贪心正好走到了 ←←←。另外注意：每一步最可能的选择，连起来不一定是最可能的序列。贪心只得到 0.061，宽度 2 的束搜索找到 →→→（0.147），宽度 3 才找到 ←←←（0.191）。

The chain rule holds exactly for any distribution and any order; there is no approximation. All an autoregressive model has to do is learn each conditional $p(x_t\mid x_{<t})$ with a network. The order does not change what can be represented, but it does change how easy each conditional is to learn, and the outcome of greedy decoding: in the "x₃ first" order greedy happens to reach ←←←. Note also that the most likely choice at each step, strung together, is not necessarily the most likely sequence. Greedy only reaches 0.061, a beam of width 2 finds →→→ (0.147), and only width 3 finds ←←← (0.191).

3动手：现场训练一个二维动作的自回归模型Play: train an autoregressive model of 2D actions live

现在让网络来学条件分布。数据是机器人的二维动作 $\mathbf a=(a_1,a_2)\in[-1,1]^2$（比如末端在桌面上的一步位移），共 800 个示教样本，取自三种设计好的分布之一：“两个峰”（左上或右下，两种做法）、“圆环”、“S 形”。每一维均匀地分成 20 个箱子（17.5 节的 binning），于是一个动作就是两个 token。

同时训练两个模型，用的是同样的数据和同样的交叉熵：

自回归：$p_\theta(a_1)\,\hl{cond}{p_\theta(a_2\mid a_1)}$。第一个 token 的分布是 20 个可学习的 logit；第二个 token 由一个 [1, 32, 32, 20] 的 tanh 网络给出，输入是 $a_1$ 所在箱子的中心（乘以 1.5）。训练时这个网络吃的是真实的 $a_1$，不是模型自己抽的，这就是 teacher forcing。
各维独立：$p_\theta(a_1)\,p_\theta(a_2)$，每一维一个独立的 softmax 头，互相看不见（第一维的头 $p_\theta(a_1)$ 和自回归模型共用，两个模型只在第二维上不同）。很多“每维输出一个分类头、一次全部输出”的策略就是这个结构。

我们知道真实分布 $P^\ast$，所以可以精确地算出模型和它之间的差距：交叉熵 $H(P^\ast,Q)=-\sum P^\ast\log_2Q$（单位 bit，4.11 节），它的下限是 $P^\ast$ 自己的熵，超出的部分就是 KL 散度。

Now let a network learn the conditionals. The data are 2D robot actions $\mathbf a=(a_1,a_2)\in[-1,1]^2$ (say, one step of end-effector displacement on a table): 800 demonstration samples from one of three designed distributions: "two modes" (upper left or lower right, two ways of doing the task), "ring", and "S-curve". Each dimension is split uniformly into 20 bins (the binning of Lesson 17.5), so an action is two tokens.

Two models train at the same time, on the same data with the same cross-entropy:

Autoregressive: $p_\theta(a_1)\,\hl{cond}{p_\theta(a_2\mid a_1)}$. The first token's distribution is 20 learnable logits; the second token comes from a [1, 32, 32, 20] tanh network whose input is the centre of $a_1$'s bin (times 1.5). During training this network is fed the true $a_1$, not one drawn by the model itself: this is teacher forcing.
Independent dimensions: $p_\theta(a_1)\,p_\theta(a_2)$, one separate softmax head per dimension, blind to each other (the first head $p_\theta(a_1)$ is shared with the autoregressive model, so the two models differ only in the second dimension). Many policies that "output one classification head per dimension, all at once" have this structure.

We know the true distribution $P^\ast$, so the gap between model and truth can be computed exactly: the cross-entropy $H(P^\ast,Q)=-\sum P^\ast\log_2Q$ (in bits, Lesson 4.11), whose floor is the entropy of $P^\ast$ itself; the excess is the KL divergence.

FIG 2自回归 vs 各维独立：谁能抓住两维之间的相关性Autoregressive vs independent: who captures the correlation between dimensions

左：动作平面，横轴 $a_1$，纵轴 $a_2$。灰点是 800 个示教样本；格子是所选模型的联合分布 $Q(a_1,a_2)$（越青概率越大，按当前最大格归一化）；琥珀点是从模型祖先采样得到的 300 个动作（先抽 $a_1$，再抽 $a_2$）。琥珀色竖虚线和顶上的把手可以左右拖动，选择右上图看的 $a_1$。右上：在这个 $a_1$ 处，模型的 $p(a_2\mid a_1)$（青色柱）和真实的条件分布（紫色阶梯线）。右下：两个模型到真实分布的交叉熵随训练步数的变化，蓝为自回归，红为各维独立，绿色虚线是下限 $H(P^\ast)$。每步一个 32 样本的 mini-batch，Adam。 Left: the action plane, $a_1$ across, $a_2$ up. Grey dots are the 800 demonstrations; the cells are the chosen model's joint $Q(a_1,a_2)$ (more cyan = more probable, scaled to the current largest cell); amber dots are 300 actions drawn from the model by ancestral sampling (first $a_1$, then $a_2$). The amber dashed vertical line and its handle at the top can be dragged sideways to choose the $a_1$ shown top right. Top right: at that $a_1$, the model's $p(a_2\mid a_1)$ (cyan bars) and the true conditional (purple step line). Bottom right: each model's cross-entropy to the truth against training steps, blue for autoregressive, red for independent, the green dashed line the floor $H(P^\ast)$. Each step is one mini-batch of 32 samples, Adam.

按“训练”，等自回归模型到真实分布的 KL 降到 0.3 bit 以下（几百步之内）。Press "Train" and wait until the autoregressive model's KL to the true distribution drops below 0.3 bits (within a few hundred steps).
在“两个峰”上训练至少 200 步，把视图切到“各维独立”：它把四成以上的概率（样本也一样）放在两个空角里，KL 一直停在 1 bit 左右。On "two modes", train for at least 200 steps and switch the view to "independent": it puts more than 40% of its probability (and so of its samples) in the two empty corners, and its KL stays at about 1 bit.
换成“圆环”，训练至少 1000 步，把切片拖到中间（$|a_1|<0.2$）：自回归模型的 $p(a_2\mid a_1)$ 有两个峰，上下各占至少 30% 的概率。Switch to "ring", train for at least 1000 steps and drag the slice to the middle ($|a_1|<0.2$): the autoregressive model's $p(a_2\mid a_1)$ has two peaks, with at least 30% of the probability above and at least 30% below.

看到了什么What you just saw

“两个峰”上，各维独立的模型把左右和上下各自学对了（两个边缘分布都对），却把它们随意组合：“左 + 下”和“右 + 上”这两个示教里从未出现的动作，各分到四分之一的概率。它损失的恰好约 1 bit：每个真实动作的概率被它减半了。自回归模型的第二个头看得见 $a_1$，所以“选了左，$a_2$ 就该向上”，KL 降到 0.1 bit 左右就停了。这个底不是 0，因为只有 800 个样本，模型在学经验分布；继续训练它还会稍稍回升，这是过拟合（16.2 节）。

On "two modes" the independent model gets left/right and up/down right separately (both marginals are correct) but combines them freely: "left + down" and "right + up", two actions that never appear in the demonstrations, get a quarter of the probability each. It loses almost exactly 1 bit: it halves the probability of every real action. The autoregressive model's second head can see $a_1$, so "having chosen left, $a_2$ should go up", and its KL stops at about 0.1 bits. That floor is not 0 because there are only 800 samples and the model learns their empirical distribution; trained longer it even creeps up a little, which is overfitting (Lesson 16.2).

4另外三个视角：怎样采样、为什么会漂、图像也能这样生成Three more views: how to sample, why it drifts, and images generated the same way

解码：温度、top-k 和 top-pDecoding: temperature, top-k and top-p

训练好的模型给出每一步的分布，生成时怎么从中取值是另一个设计选择。17.5 节比较过对单个 token 取最大、采样和取期望；对一整串 token，常用的是三个旋钮，每一步都作用在条件分布上：

温度 $\hl{tau}{\tau}$：把 logit 除以 $\tau$ 再 softmax，等价于 $q\propto p^{1/\tau}$。$\tau<1$ 让分布更尖，$\tau>1$ 让它更平，$\tau\to0$ 就是贪心。
top-k：只保留概率最大的 $k$ 个取值，重新归一化。
top-p（nucleus 采样）：按概率从大到小累加，保留累计概率刚好达到 $p$ 的那个最小集合 $\hl{nucleus}{V_p}$，其余截掉。

为什么需要截断？因为学到的模型从来不会把概率精确地压到 0：softmax 的每一格都是正的，训练也不会把尾部压干净。下图用 FIG 2 的真实条件分布，再给每一步均匀地“漏”掉一小部分概率 $\varepsilon$ 到所有箱子里，模拟一个学得还不错但尾部不干净的模型。每一步只漏 5%，但一个动作有两步，一条长序列有几百步，落到“不可能”区域的机会会逐步累积。

A trained model gives a distribution at every step; how to pick values from it during generation is a separate design choice. Lesson 17.5 compared argmax, sampling and expectation for a single token; for a whole sequence of tokens three knobs are common, each applied to the conditional at every step:

Temperature $\hl{tau}{\tau}$: divide the logits by $\tau$ before the softmax, which is $q\propto p^{1/\tau}$. $\tau<1$ sharpens the distribution, $\tau>1$ flattens it, and $\tau\to0$ is greedy.
Top-k: keep only the $k$ most probable values and renormalize.
Top-p (nucleus sampling): add up probabilities from the largest down and keep the smallest set $\hl{nucleus}{V_p}$ whose total reaches $p$; cut the rest.

Why truncate at all? Because a learned model never puts exactly zero probability anywhere: every cell of a softmax is positive, and training never cleans the tails completely. The figure takes the true conditionals of FIG 2 and "leaks" a small fraction $\varepsilon$ of each step's probability uniformly into all bins, imitating a model that has learned well but has untidy tails. Each step leaks only 5%, but an action has two steps and a long sequence hundreds, so the chances of landing in an "impossible" region add up.

FIG 3解码旋钮：多样性与可靠性的取舍Decoding knobs: trading diversity for reliability

左：解码后真正被采样的联合分布（越青概率越大）和从中抽的 400 个动作：琥珀点落在真实分布的支撑内，红点落在“空格”里（真实概率不到最大格的 1%）。右：竖虚线处的一个条件分布，紫色阶梯线是漏了 $\varepsilon$ 的模型分布，青色柱是经过温度、top-k、top-p 之后实际采样用的分布，被截掉的箱子在底部标 ✕。顺序和常见实现一致：先温度，再 top-k，再 top-p。读数里的“空格概率”和“熵”都是对整个 20 × 20 表格精确算的，不是从样本估计的。 Left: the joint distribution that is actually sampled after decoding (more cyan = more probable) and 400 actions drawn from it: amber dots lie inside the support of the true distribution, red dots in "empty" cells (true probability below 1% of the largest cell). Right: one conditional at the dashed vertical line; the purple step line is the leaky model's distribution, the cyan bars are what is actually sampled after temperature, top-k and top-p, and bins that were cut are marked ✕ at the bottom. The order follows common implementations: temperature first, then top-k, then top-p. The "empty-cell mass" and "entropy" in the readout are computed exactly over the whole 20 × 20 table, not estimated from samples.

保持 $\varepsilon\ge0.05$、不截断（top-k 关、top-p = 1），把温度调到 1.5 以上：落到空格里的概率超过 8%。With $\varepsilon\ge0.05$ and no truncation (top-k off, top-p = 1), raise the temperature to 1.5 or more: the probability of landing in empty cells exceeds 8%.
在 $\varepsilon\ge0.05$ 下找一组设置，使空格概率低于 0.5%，同时熵比真实分布的熵 $H(P^\ast)$ 少不到 0.7 bit（提示：温度 1，只用 top-p）。With $\varepsilon\ge0.05$, find settings that bring the empty-cell mass below 0.5% while the entropy stays within 0.7 bits of the true entropy $H(P^\ast)$ (hint: temperature 1 and top-p only).

在“两个峰”上把温度降到 0.3：空格概率归零，但熵比真实分布少了 1.5 bit 以上，样本都挤到各峰的中心；降到 0（贪心），每一次都输出同一个动作，熵为 0，在“两个峰”上另一种做法完全消失了。温度和截断解决的是同一个矛盾的两头：尾部里是模型的错误，头部以外又有真实的多样性。top-p 的好处是按分布的形状自适应：分布尖的时候只留一两个箱子，平的时候留很多；top-k 用固定的 $k$，在平坦处会截掉真实的可能性，在尖锐处又会放进垃圾。

On "two modes", lower the temperature to 0.3: the empty-cell mass goes to zero, but the entropy falls more than 1.5 bits below the truth and the samples crowd into the centre of each mode; at 0 (greedy) every draw is the same action, the entropy is 0, and on "two modes" the other way of doing the task has vanished. Temperature and truncation address two ends of the same tension: the tail holds the model's mistakes, while beyond the head lies real diversity. Top-p adapts to the shape of the distribution: when it is sharp only one or two bins remain, when it is flat many do; top-k uses a fixed $k$, which cuts real possibilities where the distribution is flat and lets rubbish in where it is sharp.

Exposure bias：训练时喂真值，生成时喂自己Exposure bias: trained on the truth, run on itself

Teacher forcing 让训练又快又稳：每一步的条件都是数据里的真值，各步的损失互不依赖，用因果掩码一次前向就能算完整条序列（17.3 节）。但生成时，条件是模型自己前面生成的东西。只要前面有一点偏差，后面的输入就是训练中从没见过的；在没见过的输入上模型可能出更大的错，于是偏差越滚越大。语言模型里这叫 exposure bias（模型从没“暴露”在自己的错误下），模仿学习里叫协变量偏移（covariate shift）和误差累积（22.1 节会定量推导）。

下图是一个连续状态的自回归模型：$\mathbf s_{t+1}=\mathbf s_t+f_\theta(\mathbf s_t)$，状态是末端在桌面上的位置（单位 m），专家沿一条正弦形的路径走，每步约 4.5 cm，偏离了就往回拉。12 条示教都从路径上出发，专家的回拉让每一个示教状态都在路径上（偏离不到 7 mm）。两个同样结构的网络（[2, 32, 32, 2]，tanh，同一个初始化）一起训练，只有训练数据不同：

Teacher forcing（TF）：只在示教状态上学专家的下一步。
噪声注入（NI）：把示教状态随机扰动 $\sigma_{\text{inj}}$，标签是专家在扰动后的状态上会怎么走。这需要能随时询问专家：这里专家是一个公式，所以可以；对人类示教者，可以在采集时就注入噪声（DART），或者用 22.1 节的 DAgger。

测试时，两者都从路径上的 8 个起点出发，自由地跑 64 步，每步执行时再加一点执行噪声 $\sigma_a$（真实机器人总有）。

Teacher forcing makes training fast and stable: every step is conditioned on the true values from the data, the steps' losses are independent of each other, and with a causal mask a whole sequence is computed in one forward pass (Lesson 17.3). At generation time, however, the condition is what the model itself generated earlier. Any small deviation early on means later inputs the model never saw in training; on unseen inputs it may err more, and the deviation snowballs. In language models this is called exposure bias (the model was never "exposed" to its own mistakes); in imitation learning it is covariate shift and compounding error (Lesson 22.1 derives it quantitatively).

The figure shows an autoregressive model of a continuous state: $\mathbf s_{t+1}=\mathbf s_t+f_\theta(\mathbf s_t)$, where the state is the end-effector position on a table (in metres). The expert follows a sinusoidal path at about 4.5 cm per step and pulls back toward it when off. All 12 demonstrations start on the path, and the expert's pull-back keeps every demonstrated state on it (within 7 mm). Two networks of the same structure ([2, 32, 32, 2], tanh, same initialization) train side by side and differ only in their training data:

Teacher forcing (TF): learns the expert's next step on the demonstrated states only.
Noise injection (NI): perturbs each demonstrated state by $\sigma_{\text{inj}}$ and labels it with what the expert would do at the perturbed state. That needs an expert that can be asked at any time: here the expert is a formula, so it can; with human demonstrators one injects the noise during collection (DART) or uses the DAgger of Lesson 22.1.

At test time both start from 8 points on the path and run freely for 64 steps, with a little actuation noise $\sigma_a$ added at every executed step (a real robot always has some).

FIG 4训练损失更低，跑起来却更偏：exposure biasLower training loss, yet further off when run: exposure bias

上：桌面俯视（单位 m）。紫色粗线是示教所在的路径；细线是 8 次自由运行，琥珀为 TF 模型，蓝为噪声注入模型，绿为专家本身（同样的起点、同样的执行噪声）。下：离路径的距离随步数 $t$ 的变化（8 次运行的平均，单位 mm），颜色同上。读数里的“单步误差”是在示教状态上预测的下一步与专家下一步之差，也就是 teacher-forced 训练损失衡量的东西；“平均偏离”是整个 64 步里离路径距离的平均。 Top: the table seen from above (metres). The thick purple line is the path the demonstrations lie on; the thin lines are 8 free runs, amber for the TF model, blue for the noise-injection model, green for the expert itself (same starts, same actuation noise). Bottom: distance from the path against step $t$ (mean of the 8 runs, in mm), same colours. In the readout, "one-step error" is the difference between the predicted and the expert's next step on demonstrated states, which is what the teacher-forced training loss measures; "mean deviation" is the distance from the path averaged over all 64 steps.

保持 $\sigma_{\text{inj}}\ge0.04$、$\sigma_a>0$，训练至少 1500 步：TF 的单步误差比噪声注入模型更小，平均偏离却是它的 2 倍以上。Keep $\sigma_{\text{inj}}\ge0.04$ and $\sigma_a>0$ and train for at least 1500 steps: TF's one-step error is smaller than the noise-injection model's, yet its mean deviation is more than twice as large.
按“换种子”再训练至少 1500 步：TF 跑出路径后怎么走变了（它在路径外从没被训练过），噪声注入模型仍然贴着专家，平均偏离仍不到 TF 的一半。Press "New seed" and train at least 1500 steps again: how TF behaves once off the path changes (it was never trained there), while the noise-injection model still hugs the expert, with less than half of TF's mean deviation.

别只看 teacher-forced 损失Do not judge by the teacher-forced loss

在种子 1 到 12 上（默认设置，每个训练 1500 步），TF 的单步误差都比噪声注入模型小（小 2.7 到 11 倍），平均偏离却大 2.3 到 6 倍；噪声注入模型的平均偏离在 10 到 16 mm 之间，与专家本身（9.7 mm，由执行噪声造成）相差不大。原因很简单：TF 模型只在路径上学过，一离开路径，它的输出就只是网络的随意外推，可能把它拉回来，也可能让它越走越偏；噪声注入模型在路径周围也学过“该往哪儿回”。单步损失衡量的是在数据分布上的误差，部署时要的是在自己生成的分布上的表现，两者可以相差很远。

Over seeds 1 to 12 (default settings, 1500 training steps each), TF's one-step error is always smaller than the noise-injection model's (by 2.7 to 11 times), yet its mean deviation is 2.3 to 6 times larger; the noise-injection model's mean deviation lies between 10 and 16 mm, not far above the expert's own (9.7 mm, caused by the actuation noise). The reason is simple: the TF model only learned on the path, and once off it its output is just the network's arbitrary extrapolation, which may pull it back or push it further away; the noise-injection model also learned "which way back" around the path. The one-step loss measures error on the data distribution, while deployment needs performance on the distribution the model generates itself, and the two can be far apart.

图像也是序列：PixelCNN 式的逐像素生成Images are sequences too: PixelCNN-style generation pixel by pixel

任何数据，只要定一个顺序，都能变成序列。图像按光栅顺序（从左到右、从上到下）排开，$p(\mathbf x)=\prod_i p(x_i\mid x_1,\dots,x_{i-1})$：PixelRNN 和 PixelCNN 就是这样做的。PixelCNN 用掩码卷积保证第 $i$ 个像素只看得见排在它前面的像素：卷积核里“当前像素及之后”的权重被置零，也就是一个局部的因果窗口 $\hl{mask}{\mathcal C(i)}$。叠很多层之后，窗口才逐渐变大（最初的 PixelCNN 叠再多层也有一块看不见的“盲区”，后来的 Gated PixelCNN 用竖直、水平两路卷积修好了它）。

下图的数据是 6 × 6 的二值小图，每张图恰好有一条完整的横线或竖线，共 12 张，等概率，所以数据的熵是 $\log_2 12\approx3.58$ bit。两个模型都是“一个共享的小网络，对每个像素输出它亮的概率”，输入是上下文加上像素坐标：

局部窗口：上面 2 行、左右各 2 列，加上本行左边 2 个像素，共 12 个，相当于一层 5 × 5 的掩码卷积（中心及之后的权重置零）。
完整前缀：前面所有像素（没生成的位置填 0），相当于感受野覆盖整个前缀的 PixelCNN，或者 NADE、MADE 这类模型。

Any data becomes a sequence once you fix an order. Lay out an image in raster order (left to right, top to bottom) and $p(\mathbf x)=\prod_i p(x_i\mid x_1,\dots,x_{i-1})$: this is what PixelRNN and PixelCNN do. PixelCNN uses masked convolutions so that pixel $i$ can only see the pixels before it: the kernel weights on "the current pixel and after" are zeroed, which gives a local causal window $\hl{mask}{\mathcal C(i)}$. Only stacking many layers makes the window grow (the original PixelCNN keeps a "blind spot" however many layers are stacked; the later Gated PixelCNN fixed it with separate vertical and horizontal stacks).

The data in the figure are 6 × 6 binary images, each with exactly one complete horizontal or vertical line: 12 images, equally likely, so the data entropy is $\log_2 12\approx3.58$ bits. Both models are "one shared small network that outputs, for every pixel, the probability that it is on", fed with a context plus the pixel's coordinates:

Local window: 2 rows above, 2 columns to each side, plus the 2 pixels to the left in the current row, 12 pixels in all; this is one masked 5 × 5 convolution layer (weights at the centre and after it zeroed).
Full prefix: all earlier pixels (positions not yet generated are filled with 0); this is a PixelCNN whose receptive field covers the whole prefix, or a NADE/MADE-type model.

FIG 5逐像素生成：感受野决定能不能“记得”前面画过什么Pixel by pixel: the receptive field decides whether it "remembers" what it drew

左：正在生成的图像。琥珀框是当前像素，青色格子是它的上下文（所选模型看得见的像素），琥珀框里的数字是模型给出的“亮”的概率。点击任意格子可以查看那个像素的上下文；“逐像素生成”按光栅顺序一个一个地抽。右上：所选模型的 24 个样本，绿框是合法图像（恰好一条完整的线），红框不合法。右下：每张图的负对数似然（bit，对 12 张图精确平均），琥珀为局部窗口，蓝为完整前缀，绿色虚线是数据熵 3.58 bit。合法率由每个模型的 100 个样本统计。 Left: the image being generated. The amber frame is the current pixel, the cyan cells its context (the pixels the chosen model can see), and the number in the amber frame is the model's probability that the current pixel is on. Click any cell to inspect that pixel's context; "Generate" draws the pixels one by one in raster order. Top right: 24 samples of the chosen model, green frames for valid images (exactly one complete line), red for invalid ones. Bottom right: negative log-likelihood per image (bits, averaged exactly over the 12 images), amber for the local window, blue for the full prefix, the green dashed line the data entropy of 3.58 bits. Validity rates come from 100 samples of each model.

训练到完整前缀模型的似然离 3.58 bit 不到 0.5 bit，且至少 80% 的样本合法（大约需要几千张图）。Train until the full-prefix model is within 0.5 bits of 3.58 and at least 80% of its samples are valid (a few thousand images).
在至少 3000 张图之后，把模型切到“局部窗口”：合法率停在 75% 以下。看看红框：最常见的错误是整张图一条线也没有，其次是多画了一条或画了半条：窗口看不见两行以上的地方，不知道线是否已经画过了。After at least 3000 images, switch the model to "local window": its validity rate stays below 75%. Look at the red frames: the most common mistake is an image with no line at all, followed by an extra or partial line: the window cannot see more than two rows up, so it does not know whether a line has already been drawn.

一个想法，几种看法One idea, several views

概率：链式法则把联合分布精确地分解成条件分布（FIG 1）。分类：每一步都是一个 softmax 分类器，训练就是逐步的交叉熵（FIG 2）。压缩：负对数似然就是用这个模型给数据编码所需的比特数，算术编码正是 FIG 1 那样的区间切分（4.11 节）。 动力系统：生成就是让模型在自己的输出上迭代，和一个学来的动力学模型自由仿真是一回事（FIG 4，24.1 节）。 与其他生成模型：flow（19.3 节）也给精确似然、但一次生成全部维度；VAE（19.1 节）只给下界；扩散模型（19.6 节）沿“去噪”方向而不是沿序列方向一步步生成。

Probability: the chain rule factorizes the joint exactly into conditionals (FIG 1). Classification: every step is a softmax classifier and training is a per-step cross-entropy (FIG 2). Compression: the negative log-likelihood is the number of bits needed to encode the data with this model, and arithmetic coding is exactly the interval slicing of FIG 1 (Lesson 4.11). Dynamical systems: generation iterates the model on its own outputs, just like the free simulation of a learned dynamics model (FIG 4, Lesson 24.1). Other generative models: flows (Lesson 19.3) also give exact likelihoods but generate all dimensions at once; VAEs (Lesson 19.1) only give a bound; diffusion models (Lesson 19.6) generate step by step along a "denoising" direction rather than along the sequence.

5再看数学The math

提示：把鼠标悬停在公式中带底色的符号上，图中对应的元素会被高亮（FIG 1 的最后一行、FIG 2 的条件分布柱和损失曲线、FIG 3 的解码分布和截断标记、FIG 4 的 TF 曲线、FIG 5 的上下文、FIG 6 的后验）。Tip: hover the highlighted symbols in the formulas; matching elements in the figures light up (the last row of FIG 1, the conditional bars and loss curves of FIG 2, the decoded distribution and cut marks of FIG 3, the TF curves of FIG 4, the context of FIG 5, the posterior of FIG 6).

分解与似然Factorization and likelihood

对长度为 $T$ 的序列 $\mathbf x=(x_1,\dots,x_T)$，记 $x_{<t}=(x_1,\dots,x_{t-1})$。链式法则对任何联合分布都成立：

For a sequence $\mathbf x=(x_1,\dots,x_T)$ of length $T$, write $x_{<t}=(x_1,\dots,x_{t-1})$. The chain rule holds for any joint distribution:

$$ \hl{joint}{p_\theta(\mathbf x)}=\prod_{t=1}^{T}p_\theta(x_t\mid x_{<t}),\qquad \log p_\theta(\mathbf x)=\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t}) $$

自回归模型用同一个网络（RNN、因果 Transformer、掩码卷积）把前缀 $x_{<t}$ 映射成第 $t$ 步的 logit，再 softmax。只要每个条件分布都是归一化的，乘积就自动是一个归一化的联合分布：不需要配分函数（对比 19.5 节的 EBM），也不需要变量替换的雅可比行列式（对比 19.3 节的 flow）。有条件时（机器人里几乎总是这样）只需把观测 $o$（图像、指令的 token）放进每个条件里：$p_\theta(a_{1:H}\mid o)=\prod_t p_\theta(a_t\mid a_{<t},o)$。

An autoregressive model uses one network (an RNN, a causal transformer, masked convolutions) to map the prefix $x_{<t}$ to the logits of step $t$, followed by a softmax. As long as every conditional is normalized, the product is automatically a normalized joint: no partition function is needed (compare the EBMs of Lesson 19.5), and no change-of-variables Jacobian determinant (compare the flows of Lesson 19.3). With a condition (almost always the case in robotics) the observation $o$ (image and instruction tokens) simply enters every conditional: $p_\theta(a_{1:H}\mid o)=\prod_t p_\theta(a_t\mid a_{<t},o)$.

训练：逐步交叉熵，也就是 teacher forcingTraining: per-step cross-entropy, i.e. teacher forcing

最大似然（4.6 节）就是最小化数据上的负对数似然，它按步拆开：

Maximum likelihood (Lesson 4.6) minimizes the negative log-likelihood on the data, which splits by step:

$$ \hl{nll}{\mathcal L(\theta)}=-\,\mathbb E_{\mathbf x\sim p_{\text{data}}}\Big[\sum_{t=1}^{T}\log p_\theta\big(x_t\mid \hl{tf}{x_{<t}}\big)\Big]=H(p_{\text{data}})+D_{\mathrm{KL}}\big(p_{\text{data}}\,\|\,p_\theta\big)\ \ge\ H(p_{\text{data}}) $$

关键在条件里的 $x_{<t}$：它来自数据，不是来自模型，这就是 teacher forcing。于是 $T$ 项损失可以并行算：一条序列一次前向，因果掩码保证第 $t$ 个位置只看得见 $x_{<t}$。损失的下限是数据的熵，超出部分是 KL 散度（FIG 2 的右下图画的正是这两条线）。常用的单位：每个 token 的平均 bit 数 $\mathcal L/(T\ln2)$；语言模型常报的困惑度 (perplexity) 是 $2^{b}$（$b$ 为每个 token 的平均 bit 数），可以理解为“相当于在几个等可能的选项里猜”。

FIG 2 里两个模型都是这个损失，区别只在模型族：各维独立的模型强制 $p_\theta(a_2\mid a_1)=p_\theta(a_2)$，它能达到的最好结果是两个边缘分布的乘积，此时 KL 恰为数据里两维之间的互信息 $I(a_1;a_2)$（4.11 节）。“两个峰”上，知道 $a_1$ 在哪一边就知道 $a_2$ 在哪一边，互信息约 1 bit，正是读数里那 1 bit。

The key is the $x_{<t}$ in the condition: it comes from the data, not from the model, and that is teacher forcing. So the $T$ loss terms can be computed in parallel: one forward pass per sequence, with a causal mask so that position $t$ only sees $x_{<t}$. The loss is bounded below by the data entropy, and the excess is the KL divergence (exactly the two lines in the bottom-right plot of FIG 2). Common units: average bits per token, $\mathcal L/(T\ln2)$; the perplexity that language models report is $2^{b}$ with $b$ the average bits per token, read as "as hard as guessing among that many equally likely options".

Both models in FIG 2 minimize this loss; they differ only in the model family. The independent model forces $p_\theta(a_2\mid a_1)=p_\theta(a_2)$, and the best it can reach is the product of the two marginals, where the KL equals exactly the mutual information $I(a_1;a_2)$ between the two dimensions in the data (Lesson 4.11). On "two modes", knowing which side $a_1$ is on tells you which side $a_2$ is on: the mutual information is about 1 bit, which is the 1 bit in the readout.

解码规则Decoding rules

设某一步的模型分布为 $p_k$（logit 为 $z_k$），三个旋钮依次给出实际采样用的 $q_k$（从左到右：温度、top-k、top-p；$V_K$ 是概率最大的 $K$ 个取值，$[\cdot]$ 条件成立时为 1，否则为 0）：

Let the model's distribution at one step be $p_k$ (logits $z_k$); the three knobs produce, in turn, the $q_k$ actually sampled from (left to right: temperature, top-k, top-p; $V_K$ is the set of the $K$ most probable values, and $[\cdot]$ is 1 when the condition holds and 0 otherwise):

$$ q_k\propto \exp(z_k/\hl{tau}{\tau})\propto p_k^{1/\tau},\qquad q_k\propto p_k\,[k\in V_K],\qquad q_k\propto p_k\,[k\in \hl{nucleus}{V_p}],\ \ V_p=\arg\min_{V:\ \sum_{k\in V}p_k\ge p}|V| $$

两点值得注意。第一，这些规则是逐步作用的，所以 $\prod_t q_\tau(x_t\mid x_{<t})$ 一般不等于对整个序列调温 $p(\mathbf x)^{1/\tau}/Z$：每一步的归一化常数依赖于前缀。第二，贪心解码（$\tau\to0$）和束搜索求的是“高概率序列”，而不是“典型序列”：FIG 1 里贪心不是最可能的序列；对长序列，最可能的序列往往重复、单调（语言模型里著名的“复读”），对机器人则意味着永远只走一种模式。要多样性就采样，要可靠就截尾，两者通常一起用。

Two points deserve attention. First, these rules act step by step, so $\prod_t q_\tau(x_t\mid x_{<t})$ is in general not the whole-sequence tempering $p(\mathbf x)^{1/\tau}/Z$: each step's normalizer depends on the prefix. Second, greedy decoding ($\tau\to0$) and beam search look for "high-probability sequences", not "typical sequences": in FIG 1 greedy is not the most likely sequence; for long sequences the most likely sequence is often repetitive and dull (the famous "repetition" of language models), and for a robot it means always taking one mode. For diversity sample; for reliability cut the tail; usually both are used together.

Exposure bias 的来源Where exposure bias comes from

训练最小化的是在数据前缀上的期望误差 $\mathbb E_{x_{<t}\sim p_{\text{data}}}[\ell_t]$，部署时实际承受的是 $\mathbb E_{x_{<t}\sim p_\theta}[\ell_t]$，前缀来自模型自己。两者之差取决于两个分布离得多远，而这个距离本身又随步数增长：每一步多一点偏差，后面所有步的输入分布都变了。模仿学习里，如果每步出错的概率是 $\varepsilon$，只在专家状态上训练的策略在 $T$ 步里的总代价可以按 $\varepsilon T^2$ 增长，而不是 $\varepsilon T$（Ross 与 Bagnell，2010；22.1 节推导这个界和 DAgger 的修正）。对策大致有三类：让训练分布接近部署分布（噪声注入、DART、DAgger、scheduled sampling，即训练时随机用模型自己的输出替换部分前缀）；减少需要自回归的步数（一次输出整段动作的 action chunking，22.2 节）；以及闭环：每执行几步就重新观测，让真实观测而不是模型的想象来当条件。

Training minimizes the expected error over data prefixes, $\mathbb E_{x_{<t}\sim p_{\text{data}}}[\ell_t]$; deployment actually suffers $\mathbb E_{x_{<t}\sim p_\theta}[\ell_t]$, with prefixes from the model itself. The difference depends on how far apart the two distributions are, and that distance grows with the number of steps: each step's small deviation changes the input distribution of every later step. In imitation learning, if each step errs with probability $\varepsilon$, a policy trained only on expert states can incur a total cost growing like $\varepsilon T^2$ over $T$ steps rather than $\varepsilon T$ (Ross and Bagnell, 2010; Lesson 22.1 derives the bound and DAgger's fix). Remedies fall roughly into three groups: make the training distribution resemble the deployment one (noise injection, DART, DAgger, scheduled sampling, which randomly replaces part of the prefix by the model's own outputs during training); reduce the number of autoregressive steps (action chunking, which outputs a whole stretch of actions at once, Lesson 22.2); and close the loop: observe again every few steps so that real observations, not the model's imagination, are the condition.

连续值与图像Continuous values and images

每一步的输出不一定是分类。连续值可以让网络输出一个高斯的均值和方差，或者一个高斯混合（Graves 2013 用它逐笔生成手写轨迹），PixelCNN++ 用离散化的 logistic 混合表示像素值；但单个高斯在每一步都只有一个峰，多峰性就只能靠前缀来“选边”。离散化（17.5 节）的好处是每一步本身就能有多个峰，代价是精度受箱子宽度限制、token 数变多。像素模型 $p(\mathbf x)=\prod_i p(x_i\mid \hl{mask}{\mathcal C(i)})$ 只有在 $\mathcal C(i)$ 覆盖整个前缀 $x_{<i}$ 时才等于真正的链式分解；上下文更小，就等于假设了“窗口外的像素与 $x_i$ 条件独立”，FIG 5 的局部窗口就是被这个假设骗了。

Each step's output need not be a classification. For continuous values the network can output the mean and variance of a Gaussian, or a mixture of Gaussians (Graves 2013 generated handwriting pen strokes this way), and PixelCNN++ represents pixel values with a discretized logistic mixture; but a single Gaussian has only one peak per step, so multimodality must come from the prefix "choosing a side". Discretization (Lesson 17.5) lets every step be multimodal by itself, at the cost of precision limited by the bin width and more tokens. A pixel model $p(\mathbf x)=\prod_i p(x_i\mid \hl{mask}{\mathcal C(i)})$ equals the true chain-rule factorization only when $\mathcal C(i)$ covers the whole prefix $x_{<i}$; a smaller context amounts to assuming "pixels outside the window are conditionally independent of $x_i$", and that assumption is exactly what fools the local window in FIG 5.

代价：生成是串行的The price: generation is sequential

训练可以并行（teacher forcing + 因果掩码），生成却必须一步一步来：第 $t$ 步要等第 $t-1$ 步抽完。KV 缓存（17.3 节）让每一步只算新的一行，但步数本身省不掉。一个 7 维动作按每维一个 token，就要 7 次前向；50 Hz、14 维的双臂任务，一秒就是 700 个 token（17.5 节的例子）。这正是 FAST 这样的压缩 tokenizer、action chunking，以及一次并行生成整段动作的扩散和 flow matching 动作头（19.6、19.7 节）想解决的问题。

Training can be parallel (teacher forcing + causal mask), but generation must go step by step: step $t$ waits for step $t-1$ to be drawn. The KV cache (Lesson 17.3) makes each step compute only one new row, but it cannot remove steps. A 7-dimensional action with one token per dimension costs 7 forward passes; a 14-dimensional bimanual task at 50 Hz is 700 tokens per second (the example of Lesson 17.5). This is exactly what compressing tokenizers such as FAST, action chunking, and diffusion and flow-matching action heads that generate a whole chunk in parallel (Lessons 19.6 and 19.7) set out to solve.

6机器人中的它：一段动作，绕左还是绕右In robotics: one chunk of actions, left or right

一段动作（action chunk）也是一个序列：未来 $H=10$ 步的横向偏移 $a_1,\dots,a_{10}$（每步离散成 15 个箱子）。前方有障碍，示教里一半从左绕、一半从右绕；给定绕哪边，每一步在那条路线周围有一点独立的抖动。生成这段动作有三种做法：

回归均值：每一步输出条件均值。左右一平均，正好穿过障碍（16.2 节和 17.5 节讲过的问题）。
每步独立采样：每一步从它自己的边缘分布 $p(a_t)$ 里抽，步与步之间互不知道。每一步单看都没错，连起来却左右乱跳。
自回归：$p(a_t\mid a_{<t})$。第一步抽到左边，“走左边这条路线”的后验概率 $\hl{mix}{w_t}$（给定 $a_{<t}$）立刻接近 1，后面每一步都会跟着走左边。

这里的条件分布是从两种路线的混合精确算出来的（不需要训练），碰撞概率也用动态规划精确算出。

An action chunk is a sequence too: the lateral offsets $a_1,\dots,a_{10}$ of the next $H=10$ steps (each discretized into 15 bins). There is an obstacle ahead; half the demonstrations pass it on the left, half on the right; given the side, every step jitters a little around that route, independently. There are three ways to generate the chunk:

Regress the mean: output the conditional mean at every step. Averaging left and right goes straight through the obstacle (the problem of Lessons 16.2 and 17.5).
Independent per-step sampling: draw each step from its own marginal $p(a_t)$, with the steps unaware of each other. Each step alone is fine, but together they zigzag from side to side.
Autoregressive: $p(a_t\mid a_{<t})$. Once the first step lands on the left, the posterior probability of the left route given $a_{<t}$, $\hl{mix}{w_t}$, is close to 1 and every later step follows the left route.

The conditionals here are computed exactly from the mixture of the two routes (no training needed), and the collision probabilities are computed exactly by dynamic programming.

FIG 6一段动作：自回归保持一种走法，逐步独立则左右乱跳An action chunk: autoregression keeps to one route, independent steps zigzag

机器人在底部，向上前进，横轴是横向偏移，从下到上是第 1 到第 10 步。红色方块是障碍。格子是每一步用来采样的分布（越青概率越大）：“自回归”显示最近一次采样时每一步的条件分布 $p(a_t\mid a_{<t})$，其他两种显示边缘分布。线是用当前方式采样得到的动作段（换方式时清空），绿色没碰撞，红色碰撞（某个航点落进障碍，或相邻两个航点之间的连线穿过它）；最新的一段画得最粗。紫色是均值。自回归模式下，最新一段每个航点旁的数字是后验 $w_t$（走左边的概率）。 The robot is at the bottom moving up; across is the lateral offset, bottom to top are steps 1 to 10. The red block is the obstacle. The cells are the distribution each step is sampled from (more cyan = more probable): "autoregressive" shows the conditionals $p(a_t\mid a_{<t})$ of the latest sample, the other two show the marginals. Lines are chunks sampled in the current mode (cleared when the mode changes), green without collision, red with one (a waypoint inside the obstacle, or the segment between two neighbouring waypoints crossing it); the latest chunk is drawn thickest. Purple is the mean. In autoregressive mode the numbers beside the latest chunk's waypoints are the posterior $w_t$ (the probability of the left route).

保持 P(左) 在 0.4 到 0.6 之间，分别用“每步独立”和“自回归”各抽至少 20 段：前者有 95% 到 97% 的段会碰撞，后者几乎从不碰撞（约 0.01%）。Keep P(left) between 0.4 and 0.6 and draw at least 20 chunks with "independent per step" and at least 20 with "autoregressive": 95% to 97% of the former collide, the latter almost never do (about 0.01%).

这正是 VLA 采用自回归动作 token 的一个理由：同一个 Transformer 既能处理图像和指令，又能用链式法则表示动作各维、各步之间的相关性。下表列出几种代表性的做法（各篇论文的具体 tokenizer 见 17.5 节）。

This is one reason VLAs use autoregressive action tokens: the same transformer handles images and instructions and, through the chain rule, represents the correlations between action dimensions and between steps. The table lists some representative designs (the tokenizers of these papers are described in Lesson 17.5).

工作Work	自回归地生成什么What is generated autoregressively
Decision Transformer (Chen et al., 2021)	把（目标回报，状态，动作）交替排成一条序列，用因果 Transformer 根据前面的一切预测下一个动作。Interleaves (return-to-go, state, action) into one sequence and predicts the next action from everything before it with a causal transformer.
Trajectory Transformer (Janner et al., 2021)	把状态、动作、奖励的每一维都离散化，整条轨迹逐 token 建模，再用束搜索做规划。Discretizes every dimension of states, actions and rewards, models the whole trajectory token by token, and plans with beam search.
Gato (Reed et al., 2022)	文字、图像 patch、关节和动作都变成 token，一个 Transformer 自回归地输出下一个 token，同一组权重控制多种机器人和游戏。Text, image patches, joints and actions all become tokens; one transformer outputs the next token autoregressively, and the same weights control several robots and games.
RT-2 (Brohan et al., 2023)、OpenVLA (Kim et al., 2024)	视觉语言模型把动作当成文字“说”出来：每个动作维度一个 token，按顺序逐个解码（OpenVLA 是 7 个）。A vision-language model "speaks" the action as text: one token per action dimension, decoded one after another (7 for OpenVLA).
VQ-BeT (Lee et al., 2024)	先把动作段编码成残差 VQ 的几层码字，再预测码字与连续偏移。Encodes action chunks into a few levels of residual-VQ codes, then predicts the codes and a continuous offset.
π0-FAST (Pertsch et al., 2025)	用 DCT 加 BPE 把一秒的动作压成少得多的 token（17.5 节：50 Hz 双臂任务从 700 个降到 53 个），再自回归解码，从而在高频率的任务上也能用 VLA 的自回归头。Compresses one second of actions into far fewer tokens with DCT plus BPE (Lesson 17.5: 700 down to 53 for a 50 Hz bimanual task) and decodes them autoregressively, which makes an autoregressive VLA head usable on high-frequency tasks.

真实系统里In real systems

延迟是第一约束：每个动作 token 都是整个大模型的一次前向（有 KV 缓存也一样），所以控制频率直接受 token 数限制；这也是 ACT、Diffusion Policy 这类一次输出整段动作的方法（22.2 节）以及 π0 的 flow matching 动作头（19.7 节）流行的原因之一。 解码设置要在机器人上调：语言模型的默认温度不一定适合动作；在动作 token 上，常见做法是贪心或低温解码，并用约束解码保证动作位置上只出现合法的动作 token（17.5 节的误区 4）。 闭环胜过开环：真实系统一般只执行一段动作的前几步就重新观测、重新生成，让观测而不是模型自己的预测来做条件，从而把 exposure bias 限制在很短的范围内。 维度顺序是设计选择：自回归地解码一个多维动作时，先解哪一维会影响每一步条件分布的难易和贪心解码的结果（FIG 1 的顺序实验）。

Latency is the first constraint: every action token is one forward pass of the whole large model (KV cache or not), so the control rate is directly limited by the token count; this is one reason why methods that output a whole chunk at once, such as ACT and Diffusion Policy (Lesson 22.2), and the flow-matching action head of π0 (Lesson 19.7) are popular. Tune decoding on the robot: a language model's default temperature does not necessarily suit actions; on action tokens greedy or low-temperature decoding is common, with constrained decoding so that only legal action tokens can appear at action positions (Pitfall 4 of Lesson 17.5). Closed loop beats open loop: real systems usually execute only the first few steps of a chunk, then observe and generate again, so that observations rather than the model's own predictions are the condition, which confines exposure bias to a short horizon. The order of dimensions is a design choice: when a multi-dimensional action is decoded autoregressively, which dimension comes first changes how hard each conditional is to learn and what greedy decoding returns (the order experiment of FIG 1).

7常见误区Common pitfalls

误区 1：用 teacher-forced 的验证损失评判一个要自由运行的模型Pitfall 1: judging a free-running model by its teacher-forced validation loss

FIG 4：TF 模型的单步误差更小，自由运行时却偏得更远。teacher-forced 损失只说明“在数据前缀上预测得好”，说明不了“在自己的前缀上表现好”。对要部署的策略或动力学模型，一定要做闭环评测：自由运行多步，或者直接在机器人和仿真里跑任务，看成功率和偏离。

FIG 4: the TF model has the smaller one-step error but drifts further when run freely. A teacher-forced loss only says "predicts well on data prefixes", not "behaves well on its own prefixes". For a policy or dynamics model you are going to deploy, always evaluate in closed loop: run it freely for many steps, or run the task on the robot or in simulation and look at success rates and deviations.

误区 2：每个维度或每一步各自独立采样Pitfall 2: sampling each dimension or each step independently

各维独立的分类头能学对每一维的边缘分布，却会把不同模式的维度随意拼在一起（FIG 2，“两个峰”上一半的概率落在空角）；逐步独立采样会在两条路线之间乱跳（FIG 6，P(左) = 0.5 时约 97% 碰撞）。要么让后面的维度和步以前面的为条件（自回归），要么一次从一个能表示联合分布的模型里采整段（VAE、扩散、flow matching）。

Independent classification heads learn each dimension's marginal correctly but glue together dimensions from different modes (FIG 2: on "two modes" half of the probability lands in empty corners); independent per-step sampling jumps between two routes (FIG 6: about 97% collide at P(left) = 0.5). Either condition later dimensions and steps on earlier ones (autoregression), or draw the whole chunk at once from a model of the joint (VAE, diffusion, flow matching).

误区 3：把温度、top-k、top-p 当成无害的“随机程度”旋钮Pitfall 3: treating temperature, top-k and top-p as harmless "randomness" knobs

它们改变的是你实际采样的分布，而且是逐步改变的（第 5 节）。温度太高，尾部的错误被放大（FIG 3，温度 1.5 时 8% 以上落进空格）；太低或贪心，多样性被压掉，多峰的示教只剩一种做法，并可能在长序列里反复输出同一个 token。截断参数和温度要在目标任务上联合调，并且明白似然评测（比如困惑度）是在未截断的分布上算的，调解码不会改变它。

They change the distribution you actually sample, and they change it step by step (Section 5). Too high a temperature amplifies the errors in the tail (FIG 3: above 8% in empty cells at temperature 1.5); too low, or greedy, squashes diversity, leaves only one way of doing a multimodal task, and can make a long sequence repeat the same token over and over. Tune truncation and temperature together on the target task, and remember that likelihood metrics (such as perplexity) are computed on the untruncated distribution, which decoding settings do not change.

误区 4：上下文太短，或者训练时偷看了未来Pitfall 4: too short a context, or peeking at the future during training

两种错误方向相反。上下文太短（FIG 5 的局部窗口，或只给策略最后一帧观测）相当于假设了窗口外的东西无关，模型会“忘掉”已经做过的事，比如重复画线、重复抓取。忘了因果掩码则让第 $t$ 个位置看到了 $x_t$ 本身或之后的 token，训练损失好得出奇，生成时却完全不行（17.3 节的误区 3）。检查方法：改动 $x_t$ 及之后的输入，前 $t$ 个位置的输出必须完全不变。

The two mistakes point in opposite directions. Too short a context (the local window of FIG 5, or giving a policy only the latest observation frame) assumes that whatever lies outside the window is irrelevant, and the model "forgets" what it has already done: drawing a line again, grasping again. Forgetting the causal mask lets position $t$ see $x_t$ itself or later tokens; the training loss looks suspiciously good and generation falls apart (Pitfall 3 of Lesson 17.3). Test: change the inputs at $x_t$ and later; the outputs at the first $t$ positions must not change at all.

误区 5：以为每一步最可能的选择连起来就是最可能的序列Pitfall 5: assuming the most likely choice at every step makes the most likely sequence

FIG 1：贪心选了第一步最常见的“直行”，结果整条序列只有 0.061，而最可能的序列 ←←←（0.191）第一步并不是最常见的。贪心和小宽度的束搜索都可能错过它；反过来，就算找到了最可能的序列，它也未必是好的“代表”（误区 3）。对动作序列，比起追求最可能的一条，更要紧的是整段动作内部一致、和观测一致。

FIG 1: greedy takes "straight", the most common first move, and ends with a sequence of probability only 0.061, while the most likely sequence ←←← (0.191) does not start with the most common move. Greedy and narrow beams can both miss it; conversely, even the most likely sequence is not necessarily a good "representative" (Pitfall 3). For action sequences, what matters more than finding the single most likely one is that the chunk is consistent within itself and with the observations.

8小练习Check yourself

1. 一个 7 维动作，每维 256 个箱子。用自回归模型（每维一个 token）表示它的联合分布，相比直接为所有组合建一张表，需要输出的概率个数是？1. A 7-dimensional action with 256 bins per dimension. Representing its joint distribution with an autoregressive model (one token per dimension), compared with a table over all combinations, how many probabilities must be output?

表格需要 $256^7\approx7\times10^{16}$ 格。自回归模型每一步只输出一个 256 类的 softmax，条件是前面的维度；一个动作需要 7 步，共 1792 个数。条件分布由网络根据前缀现算，不存表。选项 A 混淆了“能表示同样的分布”和“需要同样多的参数”；选项 D 的“共用一个分布”是各维独立且同分布，表示不了维度间的相关性。A table needs $256^7\approx7\times10^{16}$ cells. An autoregressive model outputs only one 256-way softmax per step, conditioned on the earlier dimensions; an action takes 7 steps, 1792 numbers in all. The conditionals are computed by the network from the prefix on the fly, not stored. Option A confuses "can represent the same distribution" with "needs as many parameters"; option D's "shared distribution" means independent and identically distributed dimensions, which cannot represent correlations between them.

2. Teacher forcing 指的是什么？2. What is teacher forcing?

它就是最大似然的直接实现：$\log p(\mathbf x)=\sum_t\log p(x_t\mid x_{<t})$ 里的 $x_{<t}$ 本来就来自数据。好处是各步可并行、训练稳定；坏处是模型从没见过自己的错误（exposure bias，FIG 4）。选项 A 更像 DAgger 或人在回路的纠正（22.1 节），选项 D 是蒸馏。It is the direct implementation of maximum likelihood: the $x_{<t}$ in $\log p(\mathbf x)=\sum_t\log p(x_t\mid x_{<t})$ comes from the data in the first place. The upside is parallel steps and stable training; the downside is that the model never sees its own mistakes (exposure bias, FIG 4). Option A is closer to DAgger or human-in-the-loop correction (Lesson 22.1); option D is distillation.

3. 某一步的模型分布是 $p=(0.5,\,0.3,\,0.15,\,0.05)$。用 top-p，$p=0.8$，实际采样的分布是？3. At one step the model's distribution is $p=(0.5,\,0.3,\,0.15,\,0.05)$. With top-p at $p=0.8$, what distribution is actually sampled?

从大到小累加：0.5，0.8，已达到 0.8，所以保留前两个，$V_p$ 有 2 个元素。重新归一化：$0.5/0.8=0.625$，$0.3/0.8=0.375$。选项 C 忘了归一化，不是一个分布。Accumulate from the largest: 0.5, 0.8, which reaches 0.8, so the first two are kept and $V_p$ has 2 elements. Renormalize: $0.5/0.8=0.625$, $0.3/0.8=0.375$. Option C forgets to renormalize and is not a distribution.

4. FIG 2 的“两个峰”上，各维独立的模型训练充分后，到真实分布的 KL 约为 1 bit。这个数字等于什么？4. On "two modes" in FIG 2, the fully trained independent model has a KL of about 1 bit to the true distribution. What does this number equal?

独立模型族里最好的是两个边缘分布的乘积 $P(a_1)P(a_2)$，而 $D_{\mathrm{KL}}(P(a_1,a_2)\,\|\,P(a_1)P(a_2))$ 正是互信息的定义（4.11 节）。“两个峰”上知道 $a_1$ 在哪边就知道 $a_2$ 在哪边，约 1 bit。它是模型族的限制，训练多久都不会消失（选项 C 错）；离散化对两个模型是一样的（选项 D 错）。The best member of the independent family is the product of the marginals $P(a_1)P(a_2)$, and $D_{\mathrm{KL}}(P(a_1,a_2)\,\|\,P(a_1)P(a_2))$ is exactly the definition of mutual information (Lesson 4.11). On "two modes" knowing the side of $a_1$ gives the side of $a_2$: about 1 bit. It is a limitation of the model family that no amount of training removes (option C is wrong); the discretization is the same for both models (option D is wrong).

5. 一个自回归动力学模型的 teacher-forced 验证损失很低，但自由仿真 50 步后偏得很远。下面哪项最不可能有帮助？5. An autoregressive dynamics model has a very low teacher-forced validation loss but drifts far after 50 steps of free simulation. Which of these is least likely to help?

问题不在数据分布上的误差，而在模型自己生成的分布上：验证损失已经很低，再降一半也只是在示教状态上更准，对示教之外的状态没有新的信息（FIG 4 里 TF 的单步误差更小却偏得更远）。选项 A、B 让训练见到偏离后的状态，选项 D 限制误差能累积的步数。The problem is not the error on the data distribution but on the distribution the model generates itself: the validation loss is already low, and halving it only makes the model more accurate on demonstrated states, adding no information about states off the demonstrations (in FIG 4 TF has the smaller one-step error yet drifts further). Options A and B show training the states after a deviation, and option D limits how many steps errors can accumulate over.

9延伸阅读Further reading

Y. Bengio, R. Ducharme, P. Vincent & C. Jauvin, A Neural Probabilistic Language Model, JMLR, 2003 — 用神经网络表示“下一个词”的条件分布a neural network for the conditional distribution of the next word
R. J. Williams & D. Zipser, A Learning Algorithm for Continually Running Fully Recurrent Neural Networks, Neural Computation, 1989 — teacher forcing 的早期来源an early source of teacher forcing
H. Larochelle & I. Murray, The Neural Autoregressive Distribution Estimator, AISTATS 2011; M. Germain, K. Gregor, I. Murray & H. Larochelle, MADE: Masked Autoencoder for Distribution Estimation, ICML 2015 — 用权重共享和掩码做自回归密度估计autoregressive density estimation with weight sharing and masks
A. Graves, Generating Sequences With Recurrent Neural Networks, arXiv, 2013 — 逐步输出高斯混合，生成连续的手写轨迹a Gaussian mixture at every step, generating continuous handwriting trajectories
A. van den Oord, N. Kalchbrenner & K. Kavukcuoglu, Pixel Recurrent Neural Networks, ICML 2016; A. van den Oord et al., Conditional Image Generation with PixelCNN Decoders, NeurIPS 2016 — PixelRNN、PixelCNN 与掩码卷积PixelRNN, PixelCNN and masked convolutions
S. Bengio, O. Vinyals, N. Jaitly & N. Shazeer, Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks, NeurIPS 2015; M. Ranzato, S. Chopra, M. Auli & W. Zaremba, Sequence Level Training with Recurrent Neural Networks, ICLR 2016 — exposure bias 及其对策exposure bias and remedies
A. Holtzman, J. Buys, L. Du, M. Forbes & Y. Choi, The Curious Case of Neural Text Degeneration, ICLR 2020 — nucleus（top-p）采样，以及为什么最可能的序列并不好nucleus (top-p) sampling, and why the most likely sequence is not good text
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei & I. Sutskever, Language Models are Unsupervised Multitask Learners, OpenAI, 2019 — GPT-2：大规模自回归 TransformerGPT-2: a large autoregressive transformer
S. Ross, G. Gordon & J. A. Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011 — DAgger：模仿学习里误差累积的分析与修正（22.1 节）DAgger: the analysis and fix of compounding error in imitation learning (Lesson 22.1)
M. Janner, Q. Li & S. Levine, Offline Reinforcement Learning as One Big Sequence Modeling Problem, NeurIPS 2021; L. Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021 — 把决策写成自回归序列建模decision making as autoregressive sequence modelling
M. J. Kim et al., OpenVLA: An Open-Source Vision-Language-Action Model, CoRL 2024 — 逐 token 解码动作的开源 VLAan open-source VLA that decodes actions token by token