PILCO、PETS 与 MBPOPILCO, PETS & MBPO

20.4 节里，SAC 让一个单关节手臂学会转到目标角度，用了大约 1000 个环境步，按 0.05 s 一步算是 50 s 的“机器人时间”；PPO 用了几千到上万步。这还只是一个关节。在 MuJoCo 的四足、人形任务上，无模型方法通常要几十万到几百万步。仿真里这只是电费，真机器人上却意味着几天的磨损、复位和看护。

无模型方法浪费的是什么？每一条转移 \((s,a,s')\) 都告诉了我们“世界怎么动”，而无模型方法只用它来改一下价值或策略，用完就基本丢了。Model-based 强化学习把这些信息存进一个动力学模型 \(\hat p(s'\mid s,a)\)（24.1 节讲了怎样学它），然后在模型里试错：想象动作序列的后果、在模型里算梯度、或者在模型里生成训练数据，只把最有希望的动作拿到真机上执行。2011 年的 PILCO 在一台真实的小车倒立摆上，只用了总共 17.5 s 的交互数据就学会了把摆甩起来并保持倒立。

代价是：模型在数据少的地方一定会错，而优化器专门会去找“模型说好、其实不好”的地方。这就是本节三个经典方法共同面对的难题，它们的不同之处在于怎样表示模型的不确定性、怎样把它往前传，以及怎样使用模型：

In Lesson 20.4, SAC taught a one-joint arm to turn to a target angle in about 1000 environment steps, which at 0.05 s per step is 50 s of "robot time"; PPO needed several thousand to tens of thousands. That is a single joint. On MuJoCo quadruped and humanoid tasks, model-free methods typically need hundreds of thousands to millions of steps. In simulation that only costs electricity; on a real robot it means days of wear, resets and supervision.

What does a model-free method waste? Every transition \((s,a,s')\) says something about how the world moves, yet a model-free method uses it to nudge a value or a policy and then mostly forgets it. Model-based reinforcement learning stores this information in a dynamics model \(\hat p(s'\mid s,a)\) (Lesson 24.1 covered how to learn it) and then experiments inside the model: it imagines the consequences of action sequences, computes gradients through the model, or generates training data in it, and only takes the most promising actions to the real robot. In 2011, PILCO learned to swing up and balance a real cart-pole with a total of 17.5 s of interaction data.

The price: the model is bound to be wrong where data are scarce, and an optimizer goes looking for exactly the places where the model says "good" and reality says otherwise. That is the problem shared by the three classic methods of this lesson. They differ in how they represent model uncertainty, how they carry it forward in time, and how they use the model:

先看一个只有一个动作参数的小问题。一台腿式机器人要选步幅 \(a\)（0 到 2 之间），目标是走得最快。真实速度先随步幅增大，在 \(a=0.95\) 附近最快（0.99），步幅超过 1.45 就会摔倒（速度记为 −0.6）。机器人一开始只谨慎地试过 6 个小步幅（0 到 0.55 之间），我们用这些数据训练一个网络，预测“步幅 → 速度”。

Model-based 的循环是：拟合模型 → 在模型里选最好的动作 → 在真机上执行 → 把结果加进数据 → 重新拟合。问题出在第二步。网络在数据之外随意外推；ReLU 网络常常顺着最后几个点的斜率一直往上画。“在模型里选最好的”就会选中外推最乐观的地方，往往是步幅 2.0，而那里机器人会摔倒。规划器越强，越能找到模型的错误：它在和模型的误差“对抗”。

补救的办法是让模型说出自己哪里没把握。用 5 个网络组成集成（21.3 节），成员在数据附近一致，在数据外各说各的。只取集成的平均并不够：成员们往往一起乐观。可以在打分时减去分歧，悲观地选择：\(\text{score}(a)=\bar\mu(a)-2\,\sigma_{\mathrm{ens}}(a)\)。

Start with a small problem that has a single action parameter. A legged robot must choose its stride \(a\) (between 0 and 2) to walk as fast as possible. The true speed first grows with the stride, peaks near \(a=0.95\) (at 0.99), and past a stride of 1.45 the robot falls (speed recorded as −0.6). At first the robot has cautiously tried only 6 short strides (between 0 and 0.55), and we train a network on them to predict "stride → speed".

The model-based loop is: fit the model → pick the best action in the model → execute it on the robot → add the outcome to the data → refit. The trouble is in step two. Outside the data the network extrapolates freely; a ReLU network often keeps drawing upward along the slope of the last few points. "Pick the best in the model" then picks wherever the extrapolation is most optimistic, often a stride of 2.0, where the robot falls. The stronger the planner, the better it finds the model's errors: it is playing against the model's mistakes.

The remedy is to let the model say where it is unsure. An ensemble of 5 networks (Lesson 21.3) agrees near the data and disagrees away from it. Taking the ensemble mean alone is not enough, since the members are often optimistic together. Instead, subtract the disagreement when scoring and choose pessimistically: \(\text{score}(a)=\bar\mu(a)-2\,\sigma_{\mathrm{ens}}(a)\).

这里 \(\hat J_k\) 是第 \(k\) 个网络预测的速度，\(\kappa=2\)。在“悲观集成”模式下，把鼠标放在公式的打分项上，下图的蓝色悲观打分曲线会加粗。三种规划器各按一次“试一次”就是一次真机试验，看看谁摔倒、谁慢慢逼近最优。

Here \(\hat J_k\) is the speed predicted by network \(k\), and \(\kappa=2\). In "pessimistic ensemble" mode, hover the score term of the formula to thicken the blue pessimistic score curve in the figure. Each press of "Try once" is one real trial; see which planner makes the robot fall and which creeps towards the optimum.

注意悲观规划器的代价：它不摔倒，但往往停在比最优略小的步幅（种子 1 的 8 次里最好 0.935，真实最优 0.99），因为它从不去试“有把握之外”的地方。乐观的规划器摔一次之后，那个数据点把模型在远处“压”了下来，之后反而学得更准：被钻的空子也是一种探索，只不过代价是一次摔倒。真机器人上，摔倒的代价往往太高，这正是三个方法都要认真对待不确定性的原因。

Note the price of pessimism: it never falls, but it tends to settle on a stride a little short of the optimum (the best of seed 1's 8 trials is 0.935, while the true optimum is 0.99), because it never tries anything outside what it is sure of. After the optimistic planner falls once, that data point pins the model down far out, and it learns more accurately afterwards: an exploited error is also a form of exploration, paid for with one fall. On a real robot a fall is often too expensive, which is why all three methods take uncertainty seriously.

本节的主要例子是经典的摆的甩起（swing-up）：角度 \(\theta\) 从竖直向下量起（\(\theta=\pi\) 为倒立），\(\ddot\theta=4a-10\sin\theta-0.1\dot\theta\)，力矩指令 \(|a|\le1\)。电机最多只能提供重力力矩的 40%，所以不能直接把摆抬上去，必须来回荡几次积累能量，再在顶端稳住。每一步 0.05 s，一次试验 100 步（5 s），每一步的代价是 \(c=\Delta\theta^2+0.1\dot\theta^2+0.01a^2\)，\(\Delta\theta\) 是离倒立的角度。最后 1 s 都在倒立 ±0.3 rad（约 17°）以内就算成功。

PETS（Chua、Calandra、McAllister 与 Levine，NeurIPS 2018）的名字就是它的三件东西：Probabilistic Ensembles with Trajectory Sampling。

• 概率集成（PE）：5 个网络，每个都像 21.3 节 FIG 2 那样输出下一步变化量的均值和方差（偶然不确定性），用高斯负对数似然训练；每个网络在自己的自助采样（bootstrap）数据上训练，成员之间的分歧就是认知不确定性。
• 轨迹采样（TS）：评估一个动作序列时，从当前状态放出 \(P\) 个粒子，每个粒子每一步从某个成员的预测高斯里采样下一状态。TS∞ 让每个粒子从头到尾跟着同一个成员；TS1 每一步重新随机挑成员。序列的分数是所有粒子代价的平均。
• CEM 规划 + MPC：用 6.11 节的交叉熵法搜索未来 \(H\) 步的动作序列：采一批序列，按上面的平均代价排名，用前几名重新拟合高斯，重复几轮。只执行第一个动作，下一步重新规划（13.4 节的滚动时域）。

每次试验结束后，把新的 100 条转移加进数据，重新训练模型，再做下一次试验。第 0 次试验用随机动作。

The main example of this lesson is the classic pendulum swing-up: the angle \(\theta\) is measured from hanging straight down (\(\theta=\pi\) is upright), \(\ddot\theta=4a-10\sin\theta-0.1\dot\theta\), with torque command \(|a|\le1\). The motor provides at most 40% of the gravity torque, so it cannot lift the pendulum directly; it has to pump energy by swinging back and forth a few times and then catch it at the top. A step is 0.05 s, a trial is 100 steps (5 s), and the cost per step is \(c=\Delta\theta^2+0.1\dot\theta^2+0.01a^2\), where \(\Delta\theta\) is the angle from upright. A trial succeeds if the last 1 s stays within ±0.3 rad (about 17°) of upright.

PETS (Chua, Calandra, McAllister and Levine, NeurIPS 2018) is named after its three parts: Probabilistic Ensembles with Trajectory Sampling.

• Probabilistic ensemble (PE): 5 networks, each outputting the mean and variance of the next-step change (aleatoric uncertainty) as in FIG 2 of Lesson 21.3, trained with the Gaussian negative log-likelihood; each network trains on its own bootstrap resample of the data, and the disagreement among members is the epistemic uncertainty.
• Trajectory sampling (TS): to score an action sequence, release \(P\) particles from the current state; at every step each particle samples its next state from the Gaussian predicted by some member. TS∞ keeps each particle with the same member from start to end; TS1 picks a random member at every step. The score of the sequence is the average cost over all particles.
• CEM planning + MPC: the cross-entropy method of Lesson 6.11 searches over action sequences \(H\) steps into the future: sample a batch of sequences, rank them by the average cost above, refit the Gaussian to the best few, and repeat a few times. Only the first action is executed, and the next step plans again (the receding horizon of Lesson 13.4).

After each trial the 100 new transitions are added to the data, the model is retrained, and the next trial begins. Trial 0 uses random actions.

几个值得注意的数字（种子 1–3，各 9 次试验）：

• 样本效率：PETS（PE + TS∞）第一次成功在第 4–7 次试验，也就是 20–35 s 的真实数据。作为对比，20.4 节的 SAC 在一个简单得多的单关节任务上也要大约 50 s。
• 最终性能：成功之后，学到的模型给出的单次代价在 425–449 之间，接近“真模型”规划的 418–439（种子 1–3，各 8 次试验；起点随机，所以真模型的代价也会变化）。在这个低维、几乎无噪声的问题上，模型足够准，model-based 方法几乎没有输给“完美模型”。
• 不确定性不是总有用：一个确定性网络（“单个确定性网络”）在种子 1–3 上都在第 4 次就成功，不比集成慢。这个摆的噪声极小、状态只有两维，几次试验后数据就把相关区域盖满了。PETS 论文里概率集成的优势出现在更难的任务上（7 自由度推物体、半猎豹奔跑），那里确定性模型会被规划器钻空子。
• 规划视野：\(H=10\)（0.5 s）时，即使用真实动力学也失败：在 0.5 s 里看不出“先往反方向荡”有什么好处。模型再准，规划器也得能看到足够远。

A few numbers worth noticing (seeds 1–3, 9 trials each):

• Sample efficiency: PETS (PE + TS∞) first succeeds on trials 4–7, that is, after 20–35 s of real data. For comparison, SAC in Lesson 20.4 needed about 50 s on a much simpler one-joint task.
• Final performance: after success, the per-trial cost with the learned model is 425–449, close to the 418–439 of planning with the true model (seeds 1–3, 8 trials each; the start is random, so even the true model's cost varies). On this low-dimensional, nearly noise-free problem the model is accurate enough that model-based control loses almost nothing to a perfect model.
• Uncertainty does not always help: a deterministic network ("single deterministic net") succeeds on trial 4 for seeds 1–3, no slower than the ensemble. This pendulum has almost no noise and only two state dimensions, and a few trials cover the relevant region. In the PETS paper, the advantage of probabilistic ensembles shows on harder tasks (a 7-DoF pusher, the half-cheetah), where deterministic models get exploited by the planner.
• Planning horizon: with \(H=10\) (0.5 s) even the true dynamics fail: within 0.5 s the planner cannot see why it should first swing the other way. However accurate the model, the planner must look far enough ahead.

一步预测的不确定性好办：集成给出几组均值和方差。难的是多步：第 2 步的输入已经是一个分布，而不是一个点，网络对分布输入没有现成的公式。办法大致有三类（PETS 论文把它们放在同一个框架里比较）：

• 只传均值（E）：每步只用预测均值，完全丢掉不确定性。
• 粒子（TS1、TS∞）：用一群样本表示分布，每个粒子单独过模型。能表示任意形状，包括“要么往左倒、要么往右倒”这种双峰分布；代价是要很多粒子。
• 矩匹配（MM）：每一步都用一个高斯（均值和协方差）概括分布，再往前推。PILCO 对高斯过程能精确算出下一步的均值和方差；对网络，PETS 的 MM 基线用粒子估计均值和协方差，再从这个高斯重新抽粒子。它简洁、平滑，但只能是单峰的。

下图把三种传法放在一起：模型是一个固定的概率集成（训练数据：2 次随机试验 + 2 次带噪声的甩起，共 400 条），摆从倒立附近 \(\theta_0=\pi+\delta\) 静止释放，力矩为 0，向前预测 2 s。

One-step uncertainty is easy: the ensemble gives several means and variances. The hard part is many steps: the input of step 2 is already a distribution, not a point, and a network has no ready formula for distributional inputs. There are roughly three approaches (the PETS paper compares them in one framework):

• Mean only (E): use only the predicted mean at each step, discarding uncertainty altogether.
• Particles (TS1, TS∞): represent the distribution by samples, each particle passing through the model on its own. Any shape can be represented, including bimodal ones like "falls either left or right"; the price is many particles.
• Moment matching (MM): summarize the distribution by one Gaussian (mean and covariance) at each step and push that forward. For a Gaussian process, PILCO computes the next mean and variance exactly; for networks, the MM baseline of PETS estimates mean and covariance from particles and redraws the particles from that Gaussian. It is compact and smooth, but always unimodal.

The figure below puts the three side by side. The model is a fixed probabilistic ensemble (trained on 2 random trials plus 2 noisy swing-ups, 400 transitions); the pendulum is released at rest near upright, \(\theta_0=\pi+\delta\), with zero torque, and predicted 2 s ahead.

为什么矩匹配会说“还立着”？倒立是不稳定平衡点：往左倒和往右倒的粒子平均起来，均值恰好留在顶上；重新从这个高斯抽粒子，又把粒子拉回中间。单峰高斯没法表示“两边都可能，中间不可能”。TS1 每步换成员，也把成员间的分歧平均掉了不少：这里 \(\delta\) 在 0.04 到 0.09 之间时，它和矩匹配一样说 20 个粒子都还立着。PETS 论文在四个任务上比较了这些传法：模型的选择（概率集成）比传法的选择重要得多，TS1 与 TS∞ 相差不大；矩匹配在低维的倒立摆上有竞争力，在半猎豹这样的高维任务上则不可靠。

但这并不说明矩匹配没用：在 PILCO 的设定里，它给出一个确定的、可微的期望代价，于是可以直接用梯度优化策略，这正是下一节的主题。

Why does moment matching say "still up"? Upright is an unstable equilibrium: averaging particles that fall left with particles that fall right leaves the mean right at the top, and redrawing particles from that Gaussian pulls them back to the middle. A unimodal Gaussian cannot express "either side is possible, the middle is not". TS1, which switches members at every step, also averages away much of the members' disagreement: here, for \(\delta\) between 0.04 and 0.09, it says just like moment matching that all 20 particles stay up. The PETS paper compares these propagation schemes on four tasks: the choice of model (a probabilistic ensemble) matters much more than the choice of propagation, TS1 and TS∞ differ little, and moment matching is competitive on the low-dimensional cart-pole but unreliable on a higher-dimensional task such as the half-cheetah.

That does not make moment matching useless: in PILCO's setting it yields a deterministic, differentiable expected cost, so the policy can be optimized directly by gradients, the topic of the next section.

PILCO（Probabilistic Inference for Learning COntrol，Deisenroth 与 Rasmussen，ICML 2011）比 PETS 早七年，思路很不一样。它有一个参数化策略 \(\pi_\theta\)，目标是让 \(T\) 步内的期望代价 \(J(\theta)=\sum_t\mathbb E[c(s_t)]\) 最小。三个关键选择：

1. 高斯过程模型：用 GP 学“状态变化量 \(\Delta=s_{t+1}-s_t\)”，每个输出维度一个 GP，平方指数核。GP 天然给出预测方差：数据附近小，远处回到先验（21.3 节的贝叶斯回归是它的有限维版本）。
2. 解析的矩匹配：如果 \(s_t\) 服从高斯，那么对平方指数核的 GP，下一步变化量的均值和协方差可以精确算出（积分有闭式解），PILCO 再用这两个矩的高斯近似 \(p(s_{t+1})\)。这样一路推下去，得到每一步状态分布的近似 \(\mathcal N(\mu_t,\Sigma_t)\)，没有任何采样。
3. 解析梯度：\(J(\theta)\) 是 \(\mu_t,\Sigma_t\) 的光滑确定函数，而 \(\mu_t,\Sigma_t\) 又是 \(\theta\) 的光滑函数。于是 \(dJ/d\theta\) 能用链式法则精确算出，交给 L-BFGS 或共轭梯度这样的确定性优化器。和 20.3 节靠采样估计、噪声很大的策略梯度相比，这是完全不同的局面。

代价函数用的是饱和代价 \(c(s)=1-\exp\!\big(-\|s-s^\star\|^2/(2\sigma_c^2)\big)\)：离目标远时代价接近 1，不再增长。它对高斯状态的期望有闭式解（第 7 节）。

下面是一个可以完整跑起来的小版本：一辆有黏性和库仑摩擦的小车（\(\ddot x=3a-0.8v-1.2\tanh 5v\)，模型看不到这个式子），要从 \(x=0\) 开到 \(x^\star=1\) 并停住，一次试验 25 步（2.5 s）。策略只有两个参数：\(a=\sin\big(w_1(x-x^\star)+w_2 v\big)\)（正弦把动作压在 \([-1,1]\) 内，PILCO 用过同样的压缩方式）。两个参数意味着我们可以把整个 \(J(w_1,w_2)\) 画成一张地图。GP 以 \((v,a)\) 为输入预测 \((\Delta x,\Delta v)\)，最多用 50 个数据点。

PILCO (Probabilistic Inference for Learning COntrol, Deisenroth and Rasmussen, ICML 2011) predates PETS by seven years and thinks quite differently. It has a parametric policy \(\pi_\theta\) and minimizes the expected cost over \(T\) steps, \(J(\theta)=\sum_t\mathbb E[c(s_t)]\). Three key choices:

1. A Gaussian-process model: a GP learns the state change \(\Delta=s_{t+1}-s_t\), one GP per output dimension, with a squared-exponential kernel. A GP gives a predictive variance for free: small near the data, back to the prior far away (the Bayesian regression of Lesson 21.3 is its finite-dimensional cousin).
2. Analytic moment matching: if \(s_t\) is Gaussian, then for a GP with a squared-exponential kernel the mean and covariance of the next change can be computed exactly (the integrals have closed forms), and PILCO approximates \(p(s_{t+1})\) by the Gaussian with these two moments. Repeating this gives an approximation \(\mathcal N(\mu_t,\Sigma_t)\) of the state distribution at every step, with no sampling at all.
3. Analytic gradients: \(J(\theta)\) is a smooth deterministic function of the \(\mu_t,\Sigma_t\), which are smooth functions of \(\theta\). So \(dJ/d\theta\) follows exactly from the chain rule and can be handed to a deterministic optimizer such as L-BFGS or conjugate gradients. Compared with the sampled, noisy policy gradients of Lesson 20.3, this is a completely different situation.

The cost is a saturating cost, \(c(s)=1-\exp\!\big(-\|s-s^\star\|^2/(2\sigma_c^2)\big)\): close to 1 far from the target, where it stops growing. Its expectation under a Gaussian state has a closed form (Section 7).

Below is a small version that runs completely: a cart with viscous and Coulomb friction (\(\ddot x=3a-0.8v-1.2\tanh 5v\), which the model never sees) must drive from \(x=0\) to \(x^\star=1\) and stop there, in trials of 25 steps (2.5 s). The policy has just two parameters, \(a=\sin\big(w_1(x-x^\star)+w_2 v\big)\) (the sine keeps the action within \([-1,1]\); PILCO used the same kind of squashing). Two parameters mean we can draw the whole \(J(w_1,w_2)\) as a map. The GP takes \((v,a)\) as input and predicts \((\Delta x,\Delta v)\), using at most 50 data points.

种子 4 的教训很典型。第 0 次随机试验的数据只覆盖了很小的速度范围，GP 在高速处回到先验均值 0，也就是“没有摩擦、没有变化”；在这个错的模型里，“越跑越快”的正反馈策略看起来不错。第一次真机试验暴露了问题，但新数据让局部梯度把策略推向另一个错误方向，梯度法只会走到最近的谷底。PILCO 是局部方法：它高度依赖初始数据和初始策略，原论文也是从几次随机试验开始的。

Seed 4's lesson is typical. The random data of trial 0 cover only a small range of speeds, and at high speeds the GP falls back to its prior mean of 0, meaning "no friction, no change"; in that wrong model a positive-feedback policy that keeps speeding up looks fine. The first real trial exposes the problem, but the new data steer the local gradient towards another wrong direction, and gradient descent only reaches the nearest valley. PILCO is a local method: it depends heavily on the initial data and policy, and the original paper also starts from a few random trials.

PILCO 的期望代价有一个有趣的性质。设预测位置 \(x\sim\mathcal N(\mu,\sigma^2)\)，饱和代价的期望是

PILCO's expected cost has an interesting property. With the predicted position \(x\sim\mathcal N(\mu,\sigma^2)\), the expected saturating cost is

离目标近时，不确定性越大期望代价越高，优化器会“求稳”；离目标很远时，均值本身已经没希望，一点运气（较大的方差）反而能让部分概率落到目标附近，期望代价下降，优化器会“愿意冒险”。Deisenroth 与 Rasmussen 把这称为饱和代价自带的探索。二次代价 \((\mu-x^\star)^2+\sigma^2\) 则永远惩罚方差。

Close to the target, more uncertainty means more expected cost and the optimizer plays safe; far from the target, the mean alone is hopeless, and a bit of luck (a larger variance) puts some probability near the target, so the expected cost falls and the optimizer is willing to gamble. Deisenroth and Rasmussen describe this as exploration built into the saturating cost. A quadratic cost \((\mu-x^\star)^2+\sigma^2\) always penalizes variance.

PETS 每一步都要做大量网络前向（本页 FIG 2 每步 3 万次），规划器就是策略，部署时也得一直规划；PILCO 的 GP 计算量随数据量的立方增长，难以扩展到高维。MBPO（Model-Based Policy Optimization，Janner、Fu、Zhang 与 Levine，NeurIPS 2019）换了一种用法：模型只负责造数据，策略学习交给 SAC（20.4 节）。

关键问题是：在模型里展开多长？从初始状态在模型里跑完整个回合，误差会一路累积（24.1 节的复合误差），策略学到的是模型的错误。MBPO 的做法是分叉展开（branched rollout）：从真实缓冲区里随机取一个真实状态，用当前策略在模型里只走 \(k\) 步（论文里很多任务 \(k=1\)，有的随训练从 1 增加到 15 或 25），把这些短轨迹放进模型缓冲区。短展开的误差小，又因为起点来自真实数据，覆盖的是策略真正会去的状态。模型数据很多，于是每个真实环境步可以做 20–40 次 SAC 更新，而普通 SAC 一般只做 1 次。

论文的理论部分给出一个下界：真实回报 \(\ge\) 分叉展开下的回报 − 一个惩罚项，惩罚里与模型误差相关的部分随 \(k\) 线性增长，与策略变化相关的部分随 \(\gamma^k\) 衰减（第 7 节）。模型越准，最优的 \(k\) 越大。

下图在 20.4 节同一个单关节任务上比较三个学习者。SAC 的代码、网络、评估都和 20.4 节完全相同；“MBPO”用 5 个概率网络的集成（结构与 FIG 2 相同，隐藏层宽 32：[4, 32, 32, 4]），前 200 步随机动作，之后每 100 步重新训练模型，每个真实步从真实状态分叉出 10 条长 \(k\) 的展开，SAC 只从模型缓冲区采样，每个真实步更新 \(G=10\) 次。

PETS runs a great many network evaluations per step (30,000 in FIG 2 on this page); the planner is the policy, so it must keep planning when deployed. PILCO's GP costs grow with the cube of the amount of data and are hard to scale to high dimensions. MBPO (Model-Based Policy Optimization, Janner, Fu, Zhang and Levine, NeurIPS 2019) uses the model differently: the model only generates data, and policy learning is left to SAC (Lesson 20.4).

The key question is how long to roll out in the model. Running whole episodes in the model from the initial state lets errors pile up (the compounding error of Lesson 24.1), and the policy learns the model's mistakes. MBPO uses branched rollouts: take a random real state from the real buffer, run the current policy in the model for only \(k\) steps (\(k=1\) on many tasks in the paper, on others growing from 1 to 15 or 25 during training), and put these short trajectories in a model buffer. Short rollouts have small errors, and since they start from real data they cover the states the policy actually visits. Model data are plentiful, so each real environment step can afford 20–40 SAC updates where ordinary SAC usually does 1.

The paper's theory gives a lower bound: true return \(\ge\) return under branched rollouts minus a penalty; the part of the penalty due to model error grows linearly in \(k\), and the part due to policy change decays like \(\gamma^k\) (Section 7). The more accurate the model, the larger the best \(k\).

The figure compares three learners on the same one-joint task as Lesson 20.4. SAC's code, networks and evaluation are exactly those of 20.4; "MBPO" uses an ensemble of 5 probabilistic networks (the same structure as FIG 2, with 32 hidden units: [4, 32, 32, 4]), takes random actions for the first 200 steps, retrains the model every 100 steps, branches 10 rollouts of length \(k\) from real states at every real step, and lets SAC sample only from the model buffer, with \(G=10\) updates per real step.

本节用代价 \(c(s,a)\)（即负奖励 \(-r\)）。三种方法都在重复同一个循环：(1) 用目前所有真实数据 \(\mathcal D\) 拟合模型 \(\hat p(s'\mid s,a)\)；(2) 在模型里改进控制器；(3) 用新控制器在真机上采数据，并入 \(\mathcal D\)。区别只在 (1) 里的不确定性表示和 (2) 的做法。

This section uses a cost \(c(s,a)\) (the negative reward \(-r\)). All three methods repeat the same loop: (1) fit a model \(\hat p(s'\mid s,a)\) to all real data \(\mathcal D\) so far; (2) improve the controller inside the model; (3) collect data on the real robot with the new controller and add it to \(\mathcal D\). They differ only in how (1) represents uncertainty and how (2) is done.

集成成员 \(b=1,\dots,B\) 各输出一个对角高斯 \(\hat p_b(s'\mid s,a)=\mathcal N\big(\mu_b(s,a),\Sigma_b(s,a)\big)\)，在自己的自助样本 \(\mathcal D_b\) 上最小化负对数似然（21.3 节）。在时刻 \(t\)，对候选动作序列 \(\mathbf a=a_{t:t+H-1}\) 放出 \(P\) 个粒子：

Ensemble members \(b=1,\dots,B\) each output a diagonal Gaussian \(\hat p_b(s'\mid s,a)=\mathcal N\big(\mu_b(s,a),\Sigma_b(s,a)\big)\) and minimize the negative log-likelihood on their own bootstrap sample \(\mathcal D_b\) (Lesson 21.3). At time \(t\), release \(P\) particles for a candidate action sequence \(\mathbf a=a_{t:t+H-1}\):

CEM 在 \(\mathbf a\) 上维护一个对角高斯，每轮采 \(N\) 个序列、取 \(\hat C\) 最小的 \(E\) 个精英重新拟合均值和方差（6.11 节），最后执行均值序列的第一个动作。论文里 \(B=5\)、\(P=20\)。平均 \(\hat C\) 是对模型不确定性的风险中性处理：成员分歧大的序列不会被额外惩罚，只是它的平均代价通常因为“有些粒子表现很差”而变高。FIG 1 的悲观打分则是风险厌恶的；离线强化学习（20.5 节讲了它的无模型方法）里也有 model-based 的做法，例如 MOPO（Yu 等，NeurIPS 2020），就是在奖励里显式减去模型的不确定性。

CEM keeps a diagonal Gaussian over \(\mathbf a\); each iteration samples \(N\) sequences, takes the \(E\) elites with the smallest \(\hat C\) and refits the mean and variance (Lesson 6.11); finally the first action of the mean sequence is executed. The paper uses \(B=5\) and \(P=20\). Averaging \(\hat C\) is a risk-neutral treatment of model uncertainty: a sequence on which the members disagree is not penalized as such; its average cost is just usually higher because some particles do badly. FIG 1's pessimistic score is risk-averse; offline RL (Lesson 20.5 covers its model-free methods) also has model-based methods, such as MOPO (Yu et al., NeurIPS 2020), which explicitly subtract the model's uncertainty from the reward.

GP 的输入是 \(\tilde x=(s,a)\)，平方指数核 \(k(\tilde x,\tilde x')=\alpha^2\exp\!\big(-\tfrac12(\tilde x-\tilde x')^\top\Lambda^{-1}(\tilde x-\tilde x')\big)\)，\(\boldsymbol\beta=(K+\sigma_n^2I)^{-1}\mathbf y\)。若输入 \(\tilde x\sim\mathcal N(\tilde\mu,\tilde\Sigma)\)，预测均值对输入分布的积分有闭式解：

The GP input is \(\tilde x=(s,a)\), with squared-exponential kernel \(k(\tilde x,\tilde x')=\alpha^2\exp\!\big(-\tfrac12(\tilde x-\tilde x')^\top\Lambda^{-1}(\tilde x-\tilde x')\big)\) and \(\boldsymbol\beta=(K+\sigma_n^2I)^{-1}\mathbf y\). If the input is \(\tilde x\sim\mathcal N(\tilde\mu,\tilde\Sigma)\), the predictive mean integrated over the input distribution has a closed form:

\(q_i\) 就是核函数被输入的不确定性“抹宽”之后的样子：\(\Lambda\) 变成 \(\tilde\Sigma+\Lambda\)。方差和输入–输出协方差也有类似的闭式解（包含一个 \(n\times n\) 的矩阵，见 Deisenroth 的博士论文），本页 FIG 4 用的就是这些公式，它们与 4 万个样本的蒙特卡洛估计在采样误差（约 1%）以内一致。把 \(\Delta\) 的矩和 \(s_t\) 合起来，得到 \(p(s_{t+1})\approx\mathcal N(\mu_{t+1},\Sigma_{t+1})\)：

\(q_i\) is the kernel "smeared out" by the input uncertainty: \(\Lambda\) becomes \(\tilde\Sigma+\Lambda\). The variance and the input–output covariance have similar closed forms (involving an \(n\times n\) matrix; see Deisenroth's PhD thesis); FIG 4 on this page uses exactly these formulas, which agree with a 40,000-sample Monte Carlo estimate to within its sampling error (about 1%). Combining the moments of \(\Delta\) with \(s_t\) gives \(p(s_{t+1})\approx\mathcal N(\mu_{t+1},\Sigma_{t+1})\):

梯度沿时间递推：\(\dfrac{dJ}{d\theta}=\sum_t\Big(\dfrac{\partial\mathbb E[c_t]}{\partial\mu_t}\dfrac{d\mu_t}{d\theta}+\dfrac{\partial\mathbb E[c_t]}{\partial\Sigma_t}\dfrac{d\Sigma_t}{d\theta}\Big)\)，其中 \(\dfrac{d\mu_t}{d\theta}=\dfrac{\partial\mu_t}{\partial\mu_{t-1}}\dfrac{d\mu_{t-1}}{d\theta}+\dfrac{\partial\mu_t}{\partial\Sigma_{t-1}}\dfrac{d\Sigma_{t-1}}{d\theta}+\dfrac{\partial\mu_t}{\partial\theta}\)（\(\Sigma_t\) 同理）。每个偏导都来自上面的闭式公式。这就是 16.3 节的反向传播沿时间展开（链式法则一层层往回乘），只是沿时间传的是“分布的矩”而不是一个状态。饱和代价的期望是

The gradient follows a recursion in time: \(\dfrac{dJ}{d\theta}=\sum_t\Big(\dfrac{\partial\mathbb E[c_t]}{\partial\mu_t}\dfrac{d\mu_t}{d\theta}+\dfrac{\partial\mathbb E[c_t]}{\partial\Sigma_t}\dfrac{d\Sigma_t}{d\theta}\Big)\), with \(\dfrac{d\mu_t}{d\theta}=\dfrac{\partial\mu_t}{\partial\mu_{t-1}}\dfrac{d\mu_{t-1}}{d\theta}+\dfrac{\partial\mu_t}{\partial\Sigma_{t-1}}\dfrac{d\Sigma_{t-1}}{d\theta}+\dfrac{\partial\mu_t}{\partial\theta}\) (and likewise for \(\Sigma_t\)). Every partial derivative comes from the closed forms above. This is the back-propagation of Lesson 16.3 unrolled through time (the chain rule multiplied back step by step), except that what travels through time is the moments of a distribution rather than a single state. The expected saturating cost is

一维时就是第 5 节的公式。对 \(\sigma^2\) 求导：\(\partial\log(1-\mathbb E[c])/\partial\sigma^2=\big((\mu-x^\star)^2-(\sigma_c^2+\sigma^2)\big)/\big(2(\sigma_c^2+\sigma^2)^2\big)\)，所以 \(|\mu-x^\star|>\sqrt{\sigma_c^2+\sigma^2}\) 时，方差越大期望代价越小。

In one dimension this is the formula of Section 5. Differentiating with respect to \(\sigma^2\): \(\partial\log(1-\mathbb E[c])/\partial\sigma^2=\big((\mu-x^\star)^2-(\sigma_c^2+\sigma^2)\big)/\big(2(\sigma_c^2+\sigma^2)^2\big)\), so when \(|\mu-x^\star|>\sqrt{\sigma_c^2+\sigma^2}\), more variance means less expected cost.

设 \(\eta[\pi]\) 为真实回报，\(\eta^{\text{branch}}[\pi]\) 为“先按采数据的策略在真实环境里走，再在某处分叉、用当前策略在模型里走 \(k\) 步”得到的回报；\(\epsilon_\pi\) 是新旧策略的最大总变差距离，\(\epsilon_{m'}\) 是模型在新策略状态分布上的总变差误差，\(r_{\max}\) 是最大奖励。MBPO 的定理 4.3 给出：

Let \(\eta[\pi]\) be the true return and \(\eta^{\text{branch}}[\pi]\) the return of "follow the data-collecting policy in the real environment, branch at some point, and run the current policy in the model for \(k\) steps"; \(\epsilon_\pi\) is the largest total-variation distance between the old and new policies, \(\epsilon_{m'}\) the model's total-variation error on the new policy's state distribution, and \(r_{\max}\) the largest reward. Theorem 4.3 of MBPO states:

前两项随 \(k\) 指数减小（在模型里走得越久，越少依赖旧策略采来的数据），最后一项随 \(k\) 线性增大（模型误差一步步累积）。只有当 \(\epsilon_{m'}\) 足够小时，最小化这个惩罚的 \(k\) 才大于 0，也就是说“用模型”才划算。论文还用实验测量了模型误差随策略偏移增长得有多快：数据越多，增长越慢，这正是支撑“短而非零的展开”的证据。

The first two terms shrink exponentially in \(k\) (the longer you run in the model, the less you rely on data from the old policy); the last grows linearly in \(k\) (model error accumulates step by step). Only when \(\epsilon_{m'}\) is small enough is the \(k\) that minimizes the penalty greater than 0, that is, only then does using the model pay. The paper also measured empirically how fast model error grows with policy shift: the more data, the slower the growth, which is the evidence behind "short but nonzero rollouts".