多任务与元学习Multi-Task & Meta-Learning

一台厨房机器人要开抽屉、倒水、擦桌子、把杯子放进洗碗机。给每个任务单独训练一个策略，每个任务分到的示教就很少；可这些任务要学的东西大部分是相同的：认出物体、控制手臂、处理接触。多任务学习（multi-task learning）用一个模型同时学所有任务，让共享的那部分从全部数据里学，每个任务自己特有的那一小部分再从自己的数据里学。这就是最早把它系统化的 Caruana（1997）的论点：相关任务的训练信号互相充当“归纳偏置”。

第二个问题更难：新任务不断出现。机械臂抓起一个没见过的工具，负载变了，重力补偿要重新标定；足式机器人走上沙地，动力学变了。我们希望它只凭几个样本、最好几秒钟就适应。21.1 节的预训练加微调是一种答案；元学习（meta-learning，“学会学习”）走得更远：在一族任务上训练，训练目标不是“在这些任务上做得好”，而是“拿到一个新任务的少量数据后，适应之后做得好”。

本节的顺序是：先看共享表示为什么省数据、什么时候反而有害（第 2 节），怎样告诉网络现在做哪个任务（第 3 节），任务之间的梯度冲突（第 4 节）；然后动手元训练 MAML 及其近亲（第 5 节），在二次问题上看清它们到底优化了什么（第 6 节），最后看“上下文学习”：把适应过程本身也交给一个网络的前向传播（第 7 节）。

A kitchen robot has to open drawers, pour water, wipe the table and load cups into the dishwasher. Train a separate policy per task and each task gets only a few demonstrations; yet most of what these tasks require is the same: recognizing objects, moving the arm, handling contact. Multi-task learning trains one model on all the tasks, so the shared part is learned from all of the data and only the small task-specific part has to come from each task's own data. That was the argument of Caruana (1997), who first made the idea systematic: the training signals of related tasks act as an "inductive bias" for one another.

The second problem is harder: new tasks keep arriving. The arm picks up a tool it has never seen, the payload changes and the gravity compensation must be recalibrated; a legged robot walks onto sand and its dynamics change. We want it to adapt from a few samples, ideally within seconds. Pre-training followed by fine-tuning (Lesson 21.1) is one answer; meta-learning ("learning to learn") goes further: it trains on a family of tasks, and the training objective is not "do well on these tasks" but "do well on a new task after adapting to a little of its data".

The plan: first why shared representations save data and when they hurt instead (Section 2), how to tell a network which task it is doing (Section 3), and gradient conflict between tasks (Section 4); then meta-training MAML and its relatives by hand (Section 5), seeing on quadratic problems what they actually optimize (Section 6), and finally "in-context learning", where the adaptation itself is handed to the forward pass of a network (Section 7).

一台机械臂里装着 6 个同型号的电机。16.2 节 FIG 3 里的关节摩擦曲线（库仑摩擦 + 低速 Stribeck 峰 + 粘性摩擦）对它们是同一个形状 \(h(v)\)，但每个电机的润滑和温度不同，曲线被放大了 \(a_t\) 倍（0.6 到 1.6 之间），力矩传感器还有一个偏置 \(b_t\)（±0.2 N·m）：电机 \(t\) 的摩擦力矩是 \(a_t\,h(v)+b_t\)。每个电机只测了 \(n=4\) 个点（噪声 σ = 0.03 N·m）。

两种学法。分开学：每个电机一个网络 \(\hl{sep}{f_{\theta_t}}(v)\)（两层隐藏层、每层 16 个 tanh 单元，321 个参数），只用自己的 4 个点。共享学（hard parameter sharing）：所有电机共用一个“主干”网络 \(\hl{trunk}{h_\phi}(v)\)，它输出 \(F\) 个特征，每个电机只有一个线性“头” \(\hat y_t=\mathbf w_t^\top h_\phi(v)+c_t\)，只有 \(F+1\) 个参数。主干用所有 24 个点训练；每个头的 \(F+1\) 个数才由自己的 4 个点决定。下图把两种学法放在一起现场训练（Adam，学习率 0.01，全批量，训练 1500 步）。

A robot arm carries 6 motors of the same model. The joint friction curve of Lesson 16.2 FIG 3 (Coulomb friction + a Stribeck peak at low speed + viscous friction) has the same shape \(h(v)\) for all of them, but each motor's lubrication and temperature differ, so its curve is scaled by \(a_t\) (between 0.6 and 1.6), and its torque sensor has an offset \(b_t\) (±0.2 N·m): motor \(t\) has friction torque \(a_t\,h(v)+b_t\). Each motor was measured at only \(n=4\) points (noise σ = 0.03 N·m).

Two ways to learn. Separately: one network \(\hl{sep}{f_{\theta_t}}(v)\) per motor (two hidden layers of 16 tanh units, 321 parameters), each using only its own 4 points. Shared (hard parameter sharing): all motors use one "trunk" network \(\hl{trunk}{h_\phi}(v)\) that outputs \(F\) features, and each motor has only a linear "head" \(\hat y_t=\mathbf w_t^\top h_\phi(v)+c_t\) with \(F+1\) parameters. The trunk is trained on all 24 points; only the \(F+1\) numbers of each head are determined by that motor's own 4 points. The figure trains both side by side, live (Adam, learning rate 0.01, full batch, 1500 steps).

默认运行里，分开训练的紫色曲线只能用 4 个点“猜”曲线的形状：两点之间要么直接连过去，要么弯得莫名其妙，低速的 Stribeck 峰基本丢了。共享的琥珀色曲线几乎和真值重合：主干从 24 个点里学到了“这一型电机的摩擦长什么样”，每个头只需要定一个比例和一个偏置。数据越少，共享的好处越大：每个电机 3 个点时误差约 0.0073 对 0.19（差 26 倍），12 个点时约 0.0003 对 0.0009（只差 3 倍）：数据够多时，每个任务自己也能学会。只有 1 个电机时共享没有任何可借的数据，\(F=1\) 的瓶颈反而只是限制（约 0.029 对 0.012）。

加入不相关的任务后，情况反过来。主干只有一个特征，它既要表示摩擦曲线，又要表示余弦形的重力力矩；梯度下降找到的折中让 6 个电机的误差涨到约 0.64，远大于分开训练的 0.024：这就是负迁移（negative transfer）。分开训练的紫色曲线完全不受影响，它们本来就互不相干。把主干加宽到 \(F=2\) 误差降到约 0.070，\(F=4\) 降到约 0.015，又比分开训练好了，但仍比没有这个任务时（\(F=4\) 约 0.005）差。按“换一组数据”试几个种子：大多数种子上都能看到同样的现象，个别种子上优化恰好找到了能兼顾两者的特征。

In the default run the separate purple curves have to "guess" the shape from 4 points: between points they either cut straight across or bend for no reason, and the low-speed Stribeck peak is mostly lost. The shared amber curves lie almost on the truth: the trunk learned "what friction looks like for this motor model" from 24 points, and each head only had to set a scale and an offset. The less data, the larger the benefit: with 3 points per motor the errors are about 0.0073 versus 0.19 (26 times apart), with 12 points about 0.0003 versus 0.0009 (only 3 times): with enough data, every task can learn on its own. With a single motor there is nothing to borrow, and the \(F=1\) bottleneck is merely a restriction (about 0.029 versus 0.012).

Add the unrelated task and the picture flips. The trunk has a single feature that must represent both the friction curve and the cosine-shaped gravity torque; the compromise gradient descent finds raises the error on the 6 motors to about 0.64, far above the 0.024 of separate training: this is negative transfer. The separate purple curves are not affected at all, since they never shared anything. Widening the trunk to \(F=2\) brings the error down to about 0.070 and \(F=4\) to about 0.015, better than separate training again, but still worse than without that task (about 0.005 at \(F=4\)). Press "New data" for a few seeds: the same thing happens on most of them; on an occasional seed the optimizer happens to find features that serve both.

FIG 1 的“每个任务一个头”需要事先知道任务的个数，新任务就得加一个新头。更常见的做法是把任务描述 \(\mathbf z\) 当作网络的另一个输入：策略 \(\pi_\theta(\mathbf a\mid\mathbf s,\mathbf z)\)，所有任务共用全部参数。\(\mathbf z\) 有几种选法：

• one-hot 任务编号 \(\mathbf z=\mathbf e_t\)：最简单，但它只是一个名字，不包含任务之间的任何关系；
• 目标（goal）：要到达的位置、目标物体的位姿，或者一张“完成后应该长什么样”的目标图像；
• 语言：“把红色的杯子放进抽屉”，用预训练的文本编码器（例如 18.3 节的 CLIP 文本塔）变成一个向量。

后两种是有意义的坐标：相近的任务有相近的 \(\mathbf z\)，网络就有机会在任务之间插值，对训练时没见过的任务“零样本”泛化。下图是一个平面上的点机器人（也可以想成末端执行器在一个平面里运动），专家的速度指令是“朝目标走，速度上限 1”。我们只在少数几个训练目标上采示教，训练两个策略（两层隐藏层、每层 32 个 tanh 单元，Adam 学习率 0.003，每步 32 个随机状态，共 2000 步）：一个输入目标坐标 \(\hl{goal}{\mathbf g}\)，一个输入 one-hot 编号。然后拖动琥珀色的测试目标，看它们能不能到达没训练过的目标。

The "one head per task" of FIG 1 needs the number of tasks up front, and a new task needs a new head. The more common approach makes a task descriptor \(\mathbf z\) another input of the network: a policy \(\pi_\theta(\mathbf a\mid\mathbf s,\mathbf z)\) whose parameters are all shared by all tasks. There are several choices of \(\mathbf z\):

• a one-hot task ID \(\mathbf z=\mathbf e_t\): the simplest, but it is just a name and carries no relation between tasks;
• a goal: a position to reach, the pose of a target object, or a goal image of "what it should look like when done";
• language: "put the red cup in the drawer", turned into a vector by a pre-trained text encoder (for instance the CLIP text tower of Lesson 18.3).

The last two are meaningful coordinates: similar tasks get similar \(\mathbf z\), so the network has a chance to interpolate between tasks and generalize "zero-shot" to tasks it never trained on. The figure shows a point robot in a plane (or an end effector moving in a plane) whose expert velocity command is "head for the goal, speed capped at 1". Demonstrations are collected for only a few training goals, and two policies are trained (two hidden layers of 32 tanh units, Adam with learning rate 0.003, 32 random states per step, 2000 steps): one takes the goal coordinates \(\hl{goal}{\mathbf g}\) as input, the other a one-hot ID. Then drag the amber test goal and see whether they can reach goals they were never trained on.

2 个训练目标时，目标条件策略在两个目标的连线上表现完美（连线中点偏差约 0.002），离开连线就差了：默认测试目标 (0.5, 0.3) 离连线 0.41，策略的偏差是 0.42，它几乎精确地走到测试目标在连线上的投影。数据只告诉了网络“目标沿这条线变化时动作怎样变”，垂直方向上它从没见过变化，于是就当作没有变化。加到 3 个不共线的目标，训练目标张成了整个平面，平均偏差从约 0.51 降到约 0.018；6 个目标约 0.007。one-hot 策略在 1、2、3、6 个目标时的平均偏差分别约为 1.04、0.65、0.50、0.35，它只能更密地“查表”，永远到不了表里没有的目标。

这正是语言条件和目标图像条件策略的逻辑：训练任务要足够多样，把描述空间“撑开”，新任务落在撑开的范围里才可能零样本成功。描述的维度越高（语言嵌入有几百维），需要的任务就越多。

With 2 training goals the goal-conditioned policy is perfect along the line through them (miss about 0.002 at the midpoint) and fails off it: the default test goal (0.5, 0.3) lies 0.41 from the line, and the policy misses it by 0.42, driving almost exactly to the test goal's projection onto the line. The data only told the network how the action changes as the goal moves along that line; it has never seen the goal change in the perpendicular direction, so it acts as if nothing changed. With 3 goals not on one line the training goals span the whole plane and the mean miss falls from about 0.51 to about 0.018; with 6 goals about 0.007. The one-hot policy's mean miss with 1, 2, 3 and 6 goals is about 1.04, 0.65, 0.50 and 0.35: all it can do is look things up in a denser table, and it never reaches a goal that is not in the table.

This is exactly the logic of language-conditioned and goal-image-conditioned policies: the training tasks must be diverse enough to "open up" the space of descriptions, and a new task can succeed zero-shot only if it falls inside what was opened up. The higher the dimension of the descriptor (language embeddings have hundreds of dimensions), the more tasks that takes.

多任务训练通常最小化加权和 \(\sum_t w_t\mathcal L_t(\boldsymbol\theta)\)，一步梯度下降走的是 \(-\eta\sum_t w_t\mathbf g_t\)。把它代入任务 1 的一阶展开：\(\Delta\mathcal L_1\approx-\eta\,(w_1\|\mathbf g_1\|^2+w_2\,\mathbf g_1^\top\mathbf g_2)\)。如果两个梯度夹角大于 90°（梯度冲突，\(\cos<0\)），而任务 2 的梯度又大得多，任务 1 的损失就会上升：一个任务的进步以另一个任务的退步为代价。

下图把这件事缩小到二维参数空间：两个二次损失 \(\mathcal L_1,\mathcal L_2\)，最小值点可以拖动。绿色曲线是 Pareto 集：在这些点上，任何移动都不可能同时降低两个损失。PCGrad（Yu 等，2020）的做法是：如果 \(\hl{g1}{\mathbf g_1}\) 和 \(\hl{g2}{\mathbf g_2}\) 冲突，就把每个梯度在对方方向上的分量去掉（投影到对方的法平面上）再相加。拖动起点、改变任务 2 的损失尺度 \(w\)，比较两种更新走到了哪里。

Multi-task training usually minimizes a weighted sum \(\sum_t w_t\mathcal L_t(\boldsymbol\theta)\), so a gradient step moves by \(-\eta\sum_t w_t\mathbf g_t\). Put that into the first-order expansion of task 1: \(\Delta\mathcal L_1\approx-\eta\,(w_1\|\mathbf g_1\|^2+w_2\,\mathbf g_1^\top\mathbf g_2)\). If the two gradients are more than 90° apart (gradient conflict, \(\cos<0\)) and task 2's gradient is much larger, task 1's loss goes up: one task's progress is paid for by another's regression.

The figure shrinks this to a two-dimensional parameter space: two quadratic losses \(\mathcal L_1,\mathcal L_2\) whose minima you can drag. The green curve is the Pareto set: at its points no move can lower both losses at once. PCGrad (Yu et al., 2020) works like this: if \(\hl{g1}{\mathbf g_1}\) and \(\hl{g2}{\mathbf g_2}\) conflict, remove from each gradient its component along the other (project it onto the other's normal plane), then add them. Drag the start, change task 2's loss scale \(w\), and compare where the two updates end up.

几点观察。(1) 冲突发生在两个最小值点“之间”的那一带：在那里两个任务想往相反方向走，这一带包括整个 Pareto 集，在 Pareto 集上两个梯度恰好反向（\(\cos=-1\)）。默认设置下，两种方法走过的 600 步里几乎每一步都在冲突。(2) 加权和总收敛到同一个点：\(\mathcal L_1+w\mathcal L_2\) 的唯一最小值点。\(w\) 从 1 增到 10，这个点沿绿色曲线滑向任务 2 的最小值，任务 1 的损失从约 0.23 涨到约 0.95，任务 2 从约 0.25 降到约 0.03。在真实的多任务网络里，\(w\) 往往不是你有意设置的，而是单位带来的：一个任务用牛顿，另一个用弧度，一个是回归，一个是交叉熵。(3) PCGrad 在这里对 \(w\) 不敏感（\(w=10\) 时任务 1 约 0.29、任务 2 约 0.21），但它也只是停在 Pareto 集上第一次碰到的地方：把起点拖到左上角或右上角，终点会沿绿色曲线跑到另一头（例如 \(w=1\) 时从 (1.8, 1.2) 出发，任务 1 约 1.13、任务 2 约 0.02）。

结论要说得谨慎：在凸问题上，每种合理的方法都停在 Pareto 集上，区别只是停在哪里；“哪个折中是对的”是你的选择，不是优化器能替你决定的。在深度网络这种非凸问题上，冲突还会把训练带进对某些任务很差的区域，PCGrad、GradNorm（Chen 等，2018）、按不确定性加权（Kendall 等，2018）这类方法报告了改进，但它们的收益随问题而变；先把各任务的损失尺度调到可比（归一化目标、按初始损失除），往往就解决了一大半。

A few observations. (1) Conflict happens in the band "between" the two minima, where the tasks want to go in opposite directions; the band contains the whole Pareto set, on which the two gradients point exactly opposite (\(\cos=-1\)). With the default settings, nearly every one of the 600 steps of both methods is a conflicting step. (2) The weighted sum always converges to the same point, the unique minimizer of \(\mathcal L_1+w\mathcal L_2\). As \(w\) goes from 1 to 10 that point slides along the green curve toward task 2's minimum; task 1's loss rises from about 0.23 to about 0.95 and task 2's falls from about 0.25 to about 0.03. In a real multi-task network \(w\) is often not something you chose but something the units chose for you: one task in newtons, another in radians, one a regression and another a cross-entropy. (3) PCGrad is insensitive to \(w\) here (task 1 about 0.29 and task 2 about 0.21 at \(w=10\)), but it, too, simply stops where it first reaches the Pareto set: drag the start to the top left or top right and its endpoint runs to the other end of the green curve (for example, from (1.8, 1.2) at \(w=1\): task 1 about 1.13, task 2 about 0.02).

The conclusion has to be stated carefully: on a convex problem every sensible method stops on the Pareto set, and they differ only in where; "which trade-off is right" is your choice, not something an optimizer can decide for you. On non-convex problems such as deep networks, conflict can also lead training into regions that are bad for some tasks, and methods such as PCGrad, GradNorm (Chen et al., 2018) and uncertainty weighting (Kendall et al., 2018) report improvements, but their benefit varies from problem to problem; first bringing the task losses to comparable scales (normalized targets, dividing by the initial loss) often solves most of it.

现在换一个问题：不是同时做好很多任务，而是拿到一个新任务后很快学会。机械臂的一个关节抓着一个未知的负载，保持关节角 \(q\) 所需的重力力矩是 \(\tau(q)=A\sin(q+\varphi)\)：幅值 \(A=mg\ell\)（N·m）取决于负载的质量和力臂，相位 \(\varphi\) 取决于负载质心相对连杆的方向。每个负载是一个任务 \(\mathcal T\)，任务分布是 \(A\in[0.5,2]\) N·m、\(\varphi\in[0,2\pi)\) 均匀分布。拿到一个新负载，机器人在 \(K=10\) 个随机姿态上测力矩（这叫支持集 \(\mathcal D^{\text{tr}}\)），然后要预测所有姿态上的力矩。

网络 \(f_{\boldsymbol\theta}(q)\) 有两层隐藏层、每层 32 个 tanh 单元（1153 个参数）。适应的方法固定为：从初始参数 \(\boldsymbol\theta\) 出发，在支持集上做几步学习率 \(\alpha=0.03\) 的普通梯度下降，\(\hl{inner}{\boldsymbol\theta'=\boldsymbol\theta-\alpha\nabla_{\boldsymbol\theta}\mathcal L^{\text{tr}}_{\mathcal T}(\boldsymbol\theta)}\)。问题是：初始参数 \(\boldsymbol\theta\) 应该取什么？

• 联合训练（“在所有任务上预训练，再微调”，21.1 节的做法）：让 \(f_{\boldsymbol\theta}\) 拟合所有负载混在一起的数据。
• MAML（Model-Agnostic Meta-Learning，Finn 等，2017）：直接优化“适应一步之后”的表现。每次抽 4 个负载，每个负载取 10 个支持点和另外 10 个查询点 \(\mathcal D^{\text{val}}\)；内循环在支持点上走一步得到 \(\boldsymbol\theta'\)，外循环用查询点上的损失 \(\mathcal L^{\text{val}}(\boldsymbol\theta')\) 对 \(\boldsymbol\theta\) 求梯度，用 Adam（学习率 0.003）更新 \(\boldsymbol\theta\)。
• 一阶 MAML（FOMAML）和 Reptile（Nichol 等，2018）：MAML 的两种便宜近似，第 6、8 节解释。

下图同时训练一个元学习模型和一个联合训练的基线，各 2000 次外循环迭代。右上是元测试：在 50 个从未见过的负载上，从各自的初始参数出发适应 0 到 10 步后的平均均方误差。

Now a different question: not doing many tasks well at once, but learning a new task quickly. One joint of a robot arm holds an unknown payload; the gravity torque needed to hold joint angle \(q\) is \(\tau(q)=A\sin(q+\varphi)\): the amplitude \(A=mg\ell\) (N·m) depends on the payload's mass and lever arm, the phase \(\varphi\) on the direction of its centre of mass relative to the link. Each payload is a task \(\mathcal T\), drawn from \(A\in[0.5,2]\) N·m and \(\varphi\in[0,2\pi)\), both uniform. Given a new payload, the robot measures the torque at \(K=10\) random poses (the support set \(\mathcal D^{\text{tr}}\)) and must then predict the torque at every pose.

The network \(f_{\boldsymbol\theta}(q)\) has two hidden layers of 32 tanh units (1153 parameters). The adaptation method is fixed: start from initial parameters \(\boldsymbol\theta\) and take a few plain gradient steps with learning rate \(\alpha=0.03\) on the support set, \(\hl{inner}{\boldsymbol\theta'=\boldsymbol\theta-\alpha\nabla_{\boldsymbol\theta}\mathcal L^{\text{tr}}_{\mathcal T}(\boldsymbol\theta)}\). The question is: what should the initial parameters \(\boldsymbol\theta\) be?

• Joint training ("pre-train on all tasks, then fine-tune", the approach of Lesson 21.1): fit \(f_{\boldsymbol\theta}\) to the data of all payloads pooled together.
• MAML (Model-Agnostic Meta-Learning, Finn et al., 2017): optimize the performance after one adaptation step directly. Each iteration draws 4 payloads, each with 10 support points and another 10 query points \(\mathcal D^{\text{val}}\); the inner loop takes one step on the support points to get \(\boldsymbol\theta'\), and the outer loop differentiates the query loss \(\mathcal L^{\text{val}}(\boldsymbol\theta')\) with respect to \(\boldsymbol\theta\) and updates \(\boldsymbol\theta\) with Adam (learning rate 0.003).
• First-order MAML (FOMAML) and Reptile (Nichol et al., 2018): two cheap approximations of MAML, explained in Sections 6 and 8.

The figure trains a meta-learned model and a jointly trained baseline side by side, 2000 outer iterations each. Top right is the meta-test: the mean squared error on 50 payloads never seen in training, after 0 to 10 adaptation steps from each model's initial parameters.

训练完以后的元测试数字（2000 次迭代，50 个测试负载上的平均均方误差；作为参照，在这 50 个负载上一直预测 0 的误差是 0.85，也就是力矩本身的平均方差）：

Meta-test numbers after training (2000 iterations, mean squared error averaged over the 50 test payloads; for reference, always predicting 0 gives 0.85 on these 50 payloads, the average variance of the torque itself):

三件值得注意的事。(1) 联合训练几乎学不到东西。相位在 \([0,2\pi)\) 上均匀分布，所有负载的平均力矩在每个姿态上都是 0，所以联合训练的模型学到的接近常数 0（误差 0.86，和一直预测 0 的 0.85 几乎一样）；更糟的是，从这个点出发微调，10 步之后反而略差（0.88）：它的隐藏单元学成了对“平均任务”最好的样子，而不是对“快速区分任务”最好的样子，甚至不如一个随机初始化（0.75）。(2) MAML 的初始参数本身是一个很差的模型（0 步 0.95，比联合训练还差），把 \(k\) 拖到 0 可以看到那条细虚线和任何负载都不像。可它只要一步梯度就降到 0.21，十步 0.068：MAML 把参数放在了一个“离所有任务都只差一步”的位置，代价是这个位置本身不对应任何任务。(3) 训练满 2000 次后，一阶近似几乎一样好（1 步 0.26 对 0.21），只是 0 步误差更大，而且学得更慢、更不平稳（右下：第 1000 次时 0.49，MAML 已是 0.22）；Reptile 起步也更慢，但最后和 MAML 持平。这和原论文的报告一致：在 MiniImagenet 少样本分类上，一阶近似的成绩几乎和完整 MAML 一样。

在这个任务族上，元学习的优势被放大了：平均任务是 0，联合训练什么有用的都学不到。如果各任务有一个很强的共同成分（例如所有负载的相位都差不多），联合训练再微调会好得多，差距会缩小：这是第 10 节误区 4 要说的。

Three things to notice. (1) Joint training learns almost nothing. The phase is uniform on \([0,2\pi)\), so the mean torque over all payloads is 0 at every pose, and the jointly trained model ends up close to the constant 0 (error 0.86, almost the same as the 0.85 of always predicting 0); worse, fine-tuning from it makes things slightly worse after 10 steps (0.88): its hidden units became what is best for the "average task", not what is best for telling tasks apart quickly, and it is even worse than a random initialization (0.75). (2) MAML's initial parameters are a poor model on their own (0.95 at 0 steps, worse than joint training); drag \(k\) to 0 and the thin dashed curve looks like no payload at all. Yet a single gradient step takes it to 0.21 and ten steps to 0.068: MAML put the parameters at a point "one step away from every task", at the cost of that point not corresponding to any task itself. (3) After 2000 iterations the first-order approximation is nearly as good (0.26 versus 0.21 after one step), with a larger 0-step error, but it learns more slowly and less steadily (bottom right: 0.49 at iteration 1000, when MAML is already at 0.22); Reptile also starts more slowly but ends level with MAML. This agrees with the original paper, which reported that on MiniImagenet few-shot classification the first-order approximation does almost as well as full MAML.

This task family magnifies the advantage of meta-learning: the average task is 0, so joint training cannot learn anything useful. If the tasks shared a strong common component (say, all payloads had similar phases), joint training plus fine-tuning would do much better and the gap would shrink: that is the point of Pitfall 4 in Section 10.

神经网络太复杂，看不清。把每个任务换成一个二维的二次损失 \(\mathcal L_t(\boldsymbol\theta)=\tfrac12(\boldsymbol\theta-\mathbf c_t)^\top A_t(\boldsymbol\theta-\mathbf c_t)\)，所有东西都能精确算出来（推导在第 8 节）：

• 联合训练最小化平均损失，起点是按曲率 \(A_t\) 加权的平均：\(\big(\sum A_t\big)^{-1}\sum A_t\mathbf c_t\)；
• MAML最小化“走一步之后”的平均损失，起点是按 \(B_t=(I-\alpha A_t)A_t(I-\alpha A_t)\) 加权的平均；
• Reptile（内循环走 \(k\) 步）的不动点按 \(I-(I-\alpha A_t)^k\) 加权。

三个任务：红色任务很“陡”（各向同性，曲率 8），绿色任务很“平”（曲率 1），蓝色任务各向异性（曲率 3 和 0.4）。拖动它们的中心，调内循环学习率 \(\alpha\) 和 Reptile 的步数 \(k\)。

Neural networks are too complicated to see through. Replace each task with a two-dimensional quadratic loss \(\mathcal L_t(\boldsymbol\theta)=\tfrac12(\boldsymbol\theta-\mathbf c_t)^\top A_t(\boldsymbol\theta-\mathbf c_t)\) and everything can be computed exactly (derivation in Section 8):

• joint training minimizes the average loss; its starting point is the curvature-weighted mean \(\big(\sum A_t\big)^{-1}\sum A_t\mathbf c_t\);
• MAML minimizes the average loss "after one step"; its starting point is the mean weighted by \(B_t=(I-\alpha A_t)A_t(I-\alpha A_t)\);
• Reptile (with \(k\) inner steps) has its fixed point at the mean weighted by \(I-(I-\alpha A_t)^k\).

Three tasks: the red task is "steep" (isotropic, curvature 8), the green one "flat" (curvature 1), the blue one anisotropic (curvatures 3 and 0.4). Drag their centres, and tune the inner learning rate \(\alpha\) and Reptile's number of steps \(k\).

默认设置（\(\alpha=0.08\)）下，联合训练的起点在 (0.65, 0.49)，被陡的红色任务拉过去：红色任务曲率最大，在平均损失里分量最重。MAML 的起点在 (−0.03, 0.19)，明显远离红色任务：从任何地方出发，在红色任务上走一步都会走掉 \(8\alpha=64\%\) 的距离，红色任务“自己就能学会”，不需要起点靠近它。结果是，适应前 MAML 起点的平均损失更大（2.88 对联合训练的 1.79），适应一步后却更小（0.61 对 0.94）：和 FIG 4 里“初始模型差、一步后好”是同一回事。\(\alpha=0.125\) 时红色任务一步就精确到达，MAML 完全不管它，一步后平均损失 0.14，联合训练 0.69。

反过来也成立：曲率很小、一步走不了多远的任务（绿色），MAML 会让起点尽量靠近它。这就是 MAML 的本质：把起点花在难以快速适应的任务上。Reptile 介于两者之间：\(k=1\) 时等于联合训练，\(k\) 越大越像“适应很多步之后”的目标，\(k=5\) 时它的不动点在 (0.31, 0.33)。

With the defaults (\(\alpha=0.08\)), joint training starts at (0.65, 0.49), pulled toward the steep red task: having the largest curvature, it weighs most in the average loss. MAML starts at (−0.03, 0.19), clearly away from the red task: from anywhere, one step on the red task covers \(8\alpha=64\%\) of the distance, so the red task "learns itself" and the starting point need not be near it. As a result, MAML's starting point has a larger mean loss before adaptation (2.88 versus 1.79 for joint training) but a smaller one after one step (0.61 versus 0.94): the same story as "poor initial model, good after one step" in FIG 4. At \(\alpha=0.125\) one step reaches the red optimum exactly, MAML ignores that task completely, and the mean loss after one step is 0.14 against joint training's 0.69.

The converse holds too: for a task with small curvature, where one step does not get far (green), MAML keeps the starting point as close as it can. That is the essence of MAML: spend the starting point on the tasks that are hard to adapt to quickly. Reptile sits in between: at \(k=1\) it equals joint training, and the larger \(k\), the more it behaves like an "after many steps" objective; at \(k=5\) its fixed point is at (0.31, 0.33).

MAML 的内循环是固定的梯度下降。为什么不让一个网络把整个“适应”也学了？给网络一个函数 \(f_{\boldsymbol\theta}(x^{*},\mathcal D)\)，它同时读入查询点 \(x^{*}\) 和整个支持集 \(\mathcal D=\{(x_i,y_i)\}\)，直接输出预测；在许多任务上用普通的监督学习训练它。部署时没有任何梯度步：新任务的数据作为上下文喂进去，一次前向传播就“学会”了。这就是上下文学习（in-context learning）：GPT-3（Brown 等，2020）在提示里放几个例子就能做新任务，是同一种机制在语言上的表现；机器人里，23.2 节的 RMA 用最近一段状态–动作历史推断地形和负载，也是在上下文里适应。

下图用最简单的一种：条件神经过程（CNP，Garnelo 等，2018）。编码器 \(e_{\boldsymbol\theta}\) 把每个上下文点 \((q_i,\tau_i)\) 变成一个 32 维向量，取平均得到任务表示 \(\hl{ctx}{\mathbf r=\tfrac1K\sum_i e_{\boldsymbol\theta}(q_i,\tau_i)}\)（取平均使它与点的顺序无关），解码器 \(\hl{cnp}{d_{\boldsymbol\theta}(q^{*},\mathbf r)}\) 输出预测。它在和 FIG 4 相同的负载分布上训练（每次 4 个负载，上下文点数在 3 到 10 之间随机，15000 次迭代），然后和 FIG 4 的元学习模型比较。为了看泛化，测试负载的幅值可以拖到 3.5 N·m，超出训练范围 [0.5, 2]。

MAML's inner loop is a fixed gradient descent. Why not let a network learn the whole "adaptation" too? Give it a function \(f_{\boldsymbol\theta}(x^{*},\mathcal D)\) that reads both the query point \(x^{*}\) and the entire support set \(\mathcal D=\{(x_i,y_i)\}\) and outputs a prediction directly; train it with ordinary supervised learning over many tasks. At deployment there is no gradient step at all: the new task's data goes in as context, and one forward pass has "learned" it. This is in-context learning: GPT-3 (Brown et al., 2020) doing a new task from a few examples placed in its prompt is the same mechanism in language; in robotics, RMA in Lesson 23.2, which infers terrain and payload from a recent window of states and actions, adapts in context too.

The figure uses the simplest kind: a conditional neural process (CNP, Garnelo et al., 2018). An encoder \(e_{\boldsymbol\theta}\) turns each context point \((q_i,\tau_i)\) into a 32-dimensional vector, and their mean is the task representation \(\hl{ctx}{\mathbf r=\tfrac1K\sum_i e_{\boldsymbol\theta}(q_i,\tau_i)}\) (the mean makes it independent of the order of the points); a decoder \(\hl{cnp}{d_{\boldsymbol\theta}(q^{*},\mathbf r)}\) outputs the prediction. It is trained on the same payload distribution as FIG 4 (4 payloads per iteration, a random number of context points between 3 and 10, 15000 iterations) and compared with FIG 4's meta-learned model. To probe generalization, the test payload's amplitude can be dragged up to 3.5 N·m, outside the training range [0.5, 2].

训练范围之内，CNP 在 50 个测试负载上的平均误差最后约 0.15 到 0.18（训练末段在约 0.12 到 0.23 之间波动），介于 MAML 适应 1 步（0.21）和 10 步（0.068）之间；它的代价是 10 次编码，加上每个查询点一次解码，没有反向传播。右图和 FIG 4 最后训练的那个模型比较；FIG 4 选 MAML 时：幅值 1 N·m 时 CNP 约 0.07 到 0.11、MAML 10 步 0.034；2 N·m 时约 0.29 对 0.12；到 3 N·m，CNP 涨到约 1.7，MAML 10 步 0.40、30 步 0.21；3.5 N·m 时约 3.4 对 0.65（10 步）。（每个幅值 20 个相位；误差本身随 \(A^2\) 增长，所以要看两者的比值。CNP 的数字在不同浏览器里略有不同：15000 次迭代会把浮点运算最后几位的微小差别放大成看得出的权重差别；MAML 的数字是可复现的。）

原因在于两者“学会”的东西不同。CNP 的适应完全由训练过的权重决定，它从没见过 3 N·m 的力矩，编码器和解码器也就不知道该输出这么大的数，再多的上下文也没用。MAML 的内循环是真正的梯度下降：只要数据要求，参数就会一直往拟合数据的方向走，即使任务在训练分布之外（Finn 和 Levine 2018 年专门讨论过这种基于梯度的元学习的泛化性）。这是一个普遍的取舍：上下文学习快、表达力强，但分布外没有保障；基于梯度的适应慢，却“至少还在学”。大模型里的上下文学习与梯度下降之间还有更深的联系，有工作表明 Transformer 在上下文里实现的就是某种梯度下降（von Oswald 等，2023）。

Inside the training range the CNP's mean error over the 50 test payloads ends at about 0.15 to 0.18 (late in training it fluctuates between about 0.12 and 0.23), between MAML after 1 step (0.21) and after 10 steps (0.068); its cost is 10 encodings plus one decoding per query point, with no backward pass. The right panel compares it with whichever model FIG 4 trained last; with MAML: at an amplitude of 1 N·m the CNP gets about 0.07 to 0.11 and MAML with 10 steps 0.034; at 2 N·m, about 0.29 versus 0.12; at 3 N·m the CNP rises to about 1.7 while MAML gets 0.40 with 10 steps and 0.21 with 30; at 3.5 N·m, about 3.4 versus 0.65 (10 steps). (20 phases per amplitude; the error itself grows with \(A^2\), so compare the ratio. The CNP's numbers differ slightly from browser to browser: 15000 iterations amplify tiny differences in the last digits of floating-point results into visibly different weights; the MAML numbers are reproducible.)

The reason is that the two "learned" different things. The CNP's adaptation is entirely determined by its trained weights: it never saw a 3 N·m torque, so neither the encoder nor the decoder knows how to output numbers that large, and more context does not help. MAML's inner loop is genuine gradient descent: as long as the data demand it, the parameters keep moving toward fitting them, even when the task lies outside the training distribution (Finn and Levine discussed this generalization of gradient-based meta-learning in 2018). It is a general trade-off: in-context learning is fast and expressive but has no guarantee out of distribution; gradient-based adaptation is slow but "at least keeps learning". In large models the link between in-context learning and gradient descent goes deeper: there is work showing that transformers can implement a form of gradient descent in context (von Oswald et al., 2023).

多任务目标与硬共享。主干参数 \(\boldsymbol\phi\)，任务 \(t\) 的头参数 \(\boldsymbol\psi_t\)，损失权重 \(w_t\)：

The multi-task objective and hard sharing. With trunk parameters \(\boldsymbol\phi\), head parameters \(\boldsymbol\psi_t\) for task \(t\) and loss weights \(w_t\):

主干的梯度是所有任务梯度的和，所以它“看到”全部 \(\sum_t n_t\) 个样本；每个头只看到自己的 \(n_t\) 个。FIG 1 里分开训练要从 4 个点定 321 个参数，共享时 6 个电机一共 \(321+6\times2=333\) 个参数（\(F=1\) 时主干正好也是 321 个，每个头 2 个），由全部 24 个点一起决定，每个头只需要 2 个数：参数个数差不多，被数据约束的程度却完全不同。Baxter（2000）给出了这件事的理论版本：任务越多，学到一个对整个任务族都好的表示所需的每任务样本数越少。

梯度冲突与 PCGrad。对 \(\sum_t w_t\mathcal L_t\) 走一步 \(\Delta\boldsymbol\theta=-\eta\sum_j w_j\mathbf g_j\)，任务 \(t\) 损失的一阶变化是

The trunk's gradient is the sum of every task's gradient, so it "sees" all \(\sum_t n_t\) samples; each head sees only its own \(n_t\). In FIG 1, separate training must pin down 321 parameters from 4 points, whereas sharing has \(321+6\times2=333\) parameters for the 6 motors (at \(F=1\) the trunk also has exactly 321, and each head 2), determined jointly by all 24 points, with each head needing just 2 numbers: about the same number of parameters, but constrained by the data in a completely different way. Baxter (2000) gave the theoretical version: the more tasks, the fewer samples per task are needed to learn a representation that is good for the whole family.

Gradient conflict and PCGrad. One step \(\Delta\boldsymbol\theta=-\eta\sum_j w_j\mathbf g_j\) on \(\sum_t w_t\mathcal L_t\) changes task \(t\)'s loss, to first order, by

右边是 PCGrad 的投影，只在两个梯度冲突（括号里的条件）时使用。投影后 \(\mathbf g_i^\top\mathbf g_j=0\)：这一步至少不会在一阶意义上直接伤害对方。两个任务的点 \(\boldsymbol\theta\) 是 Pareto 驻点，当且仅当存在 \(\lambda\in[0,1]\) 使 \(\lambda\mathbf g_1+(1-\lambda)\mathbf g_2=0\)，也就是两个梯度反向（或其中一个为零）。对 FIG 3 的二次损失，Pareto 集是 \(\boldsymbol\theta(\lambda)=\big(\lambda A_1+(1-\lambda)A_2\big)^{-1}\big(\lambda A_1\mathbf c_1+(1-\lambda)A_2\mathbf c_2\big)\)，加权和 \(\mathcal L_1+w\mathcal L_2\) 的最小值点就是 \(\lambda=1/(1+w)\) 的那一点。

MAML。内循环一步 \(\boldsymbol\theta'_{\mathcal T}=\boldsymbol\theta-\alpha\nabla\mathcal L^{\text{tr}}_{\mathcal T}(\boldsymbol\theta)\)，外循环的目标和梯度（链式法则，2.2 节；\(\nabla^2\) 是 3.5 节的 Hessian，对称）：

On the right is PCGrad's projection, applied only when the two gradients conflict (the condition in brackets). After the projection \(\mathbf g_i^\top\mathbf g_j=0\): to first order, the step at least does not directly hurt the other task. A point \(\boldsymbol\theta\) is Pareto stationary for two tasks if and only if \(\lambda\mathbf g_1+(1-\lambda)\mathbf g_2=0\) for some \(\lambda\in[0,1]\), i.e. the two gradients point in opposite directions (or one is zero). For the quadratic losses of FIG 3 the Pareto set is \(\boldsymbol\theta(\lambda)=\big(\lambda A_1+(1-\lambda)A_2\big)^{-1}\big(\lambda A_1\mathbf c_1+(1-\lambda)A_2\mathbf c_2\big)\), and the minimizer of the weighted sum \(\mathcal L_1+w\mathcal L_2\) is its point at \(\lambda=1/(1+w)\).

MAML. One inner step \(\boldsymbol\theta'_{\mathcal T}=\boldsymbol\theta-\alpha\nabla\mathcal L^{\text{tr}}_{\mathcal T}(\boldsymbol\theta)\); the outer objective and its gradient (chain rule, Lesson 2.2; \(\nabla^2\) is the Hessian of Lesson 3.5, which is symmetric):

Hessian 从不需要显式构造，只需要它乘一个向量 \(\mathbf v=\nabla\mathcal L^{\text{val}}(\boldsymbol\theta')\)。PyTorch 里对梯度再求一次梯度（create_graph=True）就得到它；FIG 4 用中心差分 \(H\mathbf v\approx\big[\nabla\mathcal L^{\text{tr}}(\boldsymbol\theta+\varepsilon\mathbf v)-\nabla\mathcal L^{\text{tr}}(\boldsymbol\theta-\varepsilon\mathbf v)\big]/(2\varepsilon)\)，\(\varepsilon=10^{-3}/\|\mathbf v\|\)，每个任务多花两次梯度计算。一阶 MAML 直接丢掉 \(\alpha\nabla^2\) 这一项，用 \(\nabla\mathcal L^{\text{val}}(\boldsymbol\theta')\) 作为元梯度。内循环走 \(k\) 步时，完整的元梯度要乘 \(k\) 个 \((I-\alpha\nabla^2\mathcal L)\)，计算和内存随 \(k\) 增长，这正是一阶方法和隐式梯度方法（iMAML，Rajeswaran 等，2019）要解决的问题。

Reptile。在任务上做 \(k\) 步 SGD 得到 \(\tilde{\boldsymbol\theta}_k\)，然后 \(\boldsymbol\theta\leftarrow\boldsymbol\theta+\beta(\tilde{\boldsymbol\theta}_k-\boldsymbol\theta)\)（FIG 4 用 \(k=5\)，把 \(\boldsymbol\theta-\tilde{\boldsymbol\theta}_k\) 当作梯度交给 Adam，每步内循环都用同样的 10 个支持点）。\(k=1\) 时 \(\tilde{\boldsymbol\theta}_1-\boldsymbol\theta=-\alpha\mathbf g\)，Reptile 就是在平均损失上做 SGD，即联合训练。\(k=2\)，两步用两个不同的小批量（梯度 \(\mathbf g_1,\mathbf g_2\)、Hessian \(H_1,H_2\)，都在 \(\boldsymbol\theta\) 处取值），做 Taylor 展开：

The Hessian is never formed explicitly; only its product with the vector \(\mathbf v=\nabla\mathcal L^{\text{val}}(\boldsymbol\theta')\) is needed. In PyTorch you get it by differentiating the gradient once more (create_graph=True); FIG 4 uses central differences \(H\mathbf v\approx\big[\nabla\mathcal L^{\text{tr}}(\boldsymbol\theta+\varepsilon\mathbf v)-\nabla\mathcal L^{\text{tr}}(\boldsymbol\theta-\varepsilon\mathbf v)\big]/(2\varepsilon)\) with \(\varepsilon=10^{-3}/\|\mathbf v\|\), two extra gradient evaluations per task. First-order MAML simply drops the \(\alpha\nabla^2\) term and uses \(\nabla\mathcal L^{\text{val}}(\boldsymbol\theta')\) as the meta-gradient. With \(k\) inner steps the full meta-gradient multiplies \(k\) factors \((I-\alpha\nabla^2\mathcal L)\), and compute and memory grow with \(k\): this is what first-order and implicit-gradient methods (iMAML, Rajeswaran et al., 2019) set out to fix.

Reptile. Run \(k\) SGD steps on a task to get \(\tilde{\boldsymbol\theta}_k\), then \(\boldsymbol\theta\leftarrow\boldsymbol\theta+\beta(\tilde{\boldsymbol\theta}_k-\boldsymbol\theta)\) (FIG 4 uses \(k=5\), hands \(\boldsymbol\theta-\tilde{\boldsymbol\theta}_k\) to Adam as the gradient, and every inner step uses the same 10 support points). For \(k=1\), \(\tilde{\boldsymbol\theta}_1-\boldsymbol\theta=-\alpha\mathbf g\): Reptile is SGD on the average loss, i.e. joint training. For \(k=2\), with two different minibatches (gradients \(\mathbf g_1,\mathbf g_2\) and Hessians \(H_1,H_2\), all evaluated at \(\boldsymbol\theta\)), a Taylor expansion gives

其中用了两个小批量可交换，\(\mathbb E[H_2\mathbf g_1]=\tfrac12\mathbb E[H_2\mathbf g_1+H_1\mathbf g_2]=\tfrac12\nabla\mathbb E[\mathbf g_1^\top\mathbf g_2]\)。Reptile 沿这个方向的反方向走：除了降低平均损失，还增大同一任务不同小批量梯度的内积，即让在一部分数据上走一步也能改善另一部分数据。Nichol 等（2018）证明 MAML 和一阶 MAML 的期望更新里也有同样两项，只是系数不同。

二次任务（FIG 5）。\(\mathcal L_t=\tfrac12(\boldsymbol\theta-\mathbf c_t)^\top A_t(\boldsymbol\theta-\mathbf c_t)\)，梯度 \(A_t(\boldsymbol\theta-\mathbf c_t)\)，走一步后 \(\boldsymbol\theta'-\mathbf c_t=(I-\alpha A_t)(\boldsymbol\theta-\mathbf c_t)\)，于是

using the exchangeability of the two minibatches, \(\mathbb E[H_2\mathbf g_1]=\tfrac12\mathbb E[H_2\mathbf g_1+H_1\mathbf g_2]=\tfrac12\nabla\mathbb E[\mathbf g_1^\top\mathbf g_2]\). Reptile moves against this direction: besides lowering the average loss, it increases the inner product between gradients from different minibatches of the same task, so that a step on one part of the data also improves another part. Nichol et al. (2018) show that the expected updates of MAML and first-order MAML contain the same two terms, only with different coefficients.

Quadratic tasks (FIG 5). With \(\mathcal L_t=\tfrac12(\boldsymbol\theta-\mathbf c_t)^\top A_t(\boldsymbol\theta-\mathbf c_t)\) the gradient is \(A_t(\boldsymbol\theta-\mathbf c_t)\), one step gives \(\boldsymbol\theta'-\mathbf c_t=(I-\alpha A_t)(\boldsymbol\theta-\mathbf c_t)\), and so

这里 \(\boldsymbol\theta^\star_{\mathrm{M}}\) 是 MAML 的起点，\(\boldsymbol\theta^\star_{\Sigma}\) 是联合训练的起点。对各向同性的 \(A_t=\lambda_tI\)，\(B_t=\lambda_t(1-\alpha\lambda_t)^2I\)：在 \(\alpha\lambda_t=1\) 时为零（一步正好到达，起点放在哪里都一样），在 \(\alpha\lambda_t\) 很小时约等于 \(\lambda_t\)（MAML 退化为联合训练，\(\alpha=0\) 时完全相同）。Reptile 的不动点满足 \(\sum_t(\tilde{\boldsymbol\theta}_{t,k}-\boldsymbol\theta)=0\)，而 \(\tilde{\boldsymbol\theta}_{t,k}-\boldsymbol\theta=-\big(I-(I-\alpha A_t)^k\big)(\boldsymbol\theta-\mathbf c_t)\)，所以 Reptile 的不动点 \(\hl{rep}{\boldsymbol\theta^\star_{\mathrm{R}}}\) 是按 \(M_t=I-(I-\alpha A_t)^k\) 加权的平均；\(k=1\) 时 \(M_t=\alpha A_t\)，与联合训练相同。

上下文学习。训练一个以整个数据集为输入的预测器（右边是 CNP 的具体形式）：

Here \(\boldsymbol\theta^\star_{\mathrm{M}}\) is MAML's starting point and \(\boldsymbol\theta^\star_{\Sigma}\) joint training's. For isotropic \(A_t=\lambda_tI\), \(B_t=\lambda_t(1-\alpha\lambda_t)^2I\): it vanishes at \(\alpha\lambda_t=1\) (one step lands exactly, so the starting point does not matter), and it is about \(\lambda_t\) when \(\alpha\lambda_t\) is small (MAML reduces to joint training, and is identical to it at \(\alpha=0\)). Reptile's fixed point satisfies \(\sum_t(\tilde{\boldsymbol\theta}_{t,k}-\boldsymbol\theta)=0\) with \(\tilde{\boldsymbol\theta}_{t,k}-\boldsymbol\theta=-\big(I-(I-\alpha A_t)^k\big)(\boldsymbol\theta-\mathbf c_t)\), so Reptile's fixed point \(\hl{rep}{\boldsymbol\theta^\star_{\mathrm{R}}}\) is the mean weighted by \(M_t=I-(I-\alpha A_t)^k\); at \(k=1\), \(M_t=\alpha A_t\), the same as joint training.

In-context learning. Train a predictor that takes a whole dataset as input (on the right, the specific form of a CNP):

用平方损失时，这个目标的最优解是后验预测均值 \(\mathbb E[y^{*}\mid x^{*},\mathcal D]\)，期望取在任务分布 \(p(\mathcal T)\) 下（平方误差的最小值点是条件均值，和 19.6 节去噪器的道理相同）：上下文学习器学到的是对这个任务族的贝叶斯推断的一个近似。这也说明了 FIG 6 的分布外失败：\(p(\mathcal T)\) 里没有 3 N·m 的负载，这样的上下文在训练目标里根本不出现，网络在那里输出什么都不受约束。

With the squared loss the optimum of this objective is the posterior predictive mean \(\mathbb E[y^{*}\mid x^{*},\mathcal D]\), the expectation taken under the task distribution \(p(\mathcal T)\) (a squared error is minimized by the conditional mean, the same fact behind the denoiser of Lesson 19.6): an in-context learner learns an approximation of Bayesian inference for this family of tasks. This also explains the out-of-distribution failure in FIG 6: \(p(\mathcal T)\) contains no 3 N·m payloads, such contexts never appear in the training objective, and nothing constrains what the network outputs there.

多任务、语言条件的模仿学习是今天操作策略的主流形式。BC-Z（Jang 等，CoRL 2021）用一条语言指令或一段人类示范视频作为任务条件，并在一部分训练中没出现的任务上零样本成功。RT-1（Brohan 等，2022）在 13 台 Everyday Robots 机器人、17 个月采集的约 13 万条示教上训练，覆盖 700 多条指令；指令用 Universal Sentence Encoder 编码，通过 FiLM 层注入 EfficientNet 图像编码器（第 3 节的 FiLM）。Octo（Octo Model Team，RSS 2024）既可以用语言、也可以用目标图像作为条件。这些系统都押注于 FIG 1 和 FIG 2 的逻辑：任务越多样，共享的视觉与运动表示越好，新指令越可能落在已覆盖的范围里。

目标条件强化学习：UVFA（Schaul 等，2015）把价值函数写成 \(V(\mathbf s,\mathbf g)\)，HER（Andrychowicz 等，2017）把一条没到达目标的轨迹“重新标注”成到达了它实际到达的那个点：一条失败轨迹变成了另一个任务的成功示范，目标空间被免费地“撑开”。Lynch 等（CoRL 2019）用同样的事后重标注，从遥操作的“自由玩耍”数据学目标条件策略。

基准：Meta-World（Yu 等，CoRL 2019）在仿真的 Sawyer 机械臂上提供 50 个操作任务，MT10/MT50 用来评测多任务学习，ML1/ML10/ML45 用来评测元学习（在训练任务上元训练，在留出的任务上测试适应）。

元学习用于快速适应：Nagabandi 等（ICLR 2019）元学习一个动力学模型，让一台真实的小型足式机器人在几步之内在线适应地形变化、腿部损伤等扰动；单样本模仿学习（Duan 等，NeurIPS 2017）把一条示范作为上下文；Finn 等（CoRL 2017）用 MAML 让机器人从单条视觉示范学会新的放置任务。PEARL（Rakelly 等，ICML 2019）推断一个任务隐变量，是强化学习里的上下文适应；23.2 节的 RMA 用历史编码器在线估计地形与负载，属于同一类。

Multi-task, language-conditioned imitation learning is the dominant form of manipulation policy today. BC-Z (Jang et al., CoRL 2021) conditions on a language instruction or a video of a human doing the task, and succeeded zero-shot on some tasks absent from training. RT-1 (Brohan et al., 2022) was trained on about 130k demonstrations collected over 17 months on 13 Everyday Robots machines, covering over 700 instructions; the instruction is encoded by the Universal Sentence Encoder and injected into an EfficientNet image encoder through FiLM layers (the FiLM of Section 3). Octo (Octo Model Team, RSS 2024) can be conditioned on either language or a goal image. All of these bet on the logic of FIG 1 and FIG 2: the more diverse the tasks, the better the shared visual and motor representation, and the more likely a new instruction falls inside the covered range.

Goal-conditioned reinforcement learning: UVFA (Schaul et al., 2015) writes the value function as \(V(\mathbf s,\mathbf g)\), and HER (Andrychowicz et al., 2017) "relabels" a trajectory that missed its goal as one that reached the point it actually reached: a failure becomes a successful demonstration of another task, and the goal space is "opened up" for free. Lynch et al. (CoRL 2019) used the same hindsight relabeling to learn goal-conditioned policies from teleoperated "play" data.

Benchmarks: Meta-World (Yu et al., CoRL 2019) provides 50 manipulation tasks on a simulated Sawyer arm, with MT10/MT50 for multi-task learning and ML1/ML10/ML45 for meta-learning (meta-train on training tasks, test adaptation on held-out ones).

Meta-learning for fast adaptation: Nagabandi et al. (ICLR 2019) meta-learned a dynamics model so that a real small legged robot adapts online, within a few steps, to perturbations such as terrain changes and a damaged leg; one-shot imitation learning (Duan et al., NeurIPS 2017) takes a single demonstration as context; Finn et al. (CoRL 2017) used MAML to let a robot learn a new placing task from a single visual demonstration. PEARL (Rakelly et al., ICML 2019) infers a latent task variable, in-context adaptation for reinforcement learning; RMA in Lesson 23.2, which estimates terrain and payload online with a history encoder, belongs to the same family.