Part VI · 第 24 章 Model-based 方法 · 24.4Part VI · Ch. 24 Model-Based Methods · 24.4

学习模型 + MPCLearned Models + MPC

MPC 的一切都建立在模型上，而模型可以从数据里学。把一个学到的动力学模型放进 MPC，要回答四个问题：用梯度还是采样去优化这个模型；能不能把非线性模型“抬”到一个线性的空间里，让每个周期仍然只解一个凸 QP（Koopman）；怎样让规划器知道模型在哪里没把握，并在代价和约束里为此留出余量（高斯过程 MPC）；以及怎样只学物理模型漏掉的那一部分（残差学习）。本节在浏览器里现场训练模型，用倒立摆、刚度渐增的弹簧、泥地里的小车、靠墙的小车和高速飞圆圈的“无人机”逐一演示，最后讨论什么时候“经典 MPC + 学到的残差”比端到端强化学习更合适。 Everything MPC does rests on its model, and the model can be learned from data. Putting a learned dynamics model inside MPC raises four questions: should you optimize through the model with gradients or by sampling; can a nonlinear model be "lifted" into a space where it is linear, so that each cycle is still a single convex QP (Koopman); how does the planner learn where the model is unsure, and leave a margin for it in the cost and in the constraints (Gaussian-process MPC); and how do you learn only the part that the physics model misses (residual learning)? This lesson trains models live in your browser and shows each idea on a pendulum, a stiffening spring, a rover crossing muddy ground, a cart near a wall and a "drone" flying fast circles, and ends with when "classical MPC plus a learned residual" is a better choice than end-to-end reinforcement learning.

约 50 分钟~50 min 前置：6.11 无梯度与采样优化 · 13.4 模型预测控制 MPC · 24.1 学习动力学模型Prereqs: 6.11 Derivative-free & sampling-based optimization · 13.4 Model predictive control · 24.1 Learning dynamics models 6 个交互6 interactives

1为什么机器人需要它Why robots need this

13.4 节的 MPC 每个周期用模型预测未来几秒、在约束下优化、只执行第一步。它默认模型是写出来的：小车是质点，自行车模型有轴距，四足是单刚体。可很多机器人身上，最要紧的那部分动力学恰恰写不出来或写不准：

空气动力：四旋翼以 10 m/s 以上飞行时，阻力、桨叶的气动效应以及旋翼下洗流与机身的相互作用会造成明显的跟踪误差，它们与速度、姿态和转速都有关，很难用几个参数描述。
轮胎和地面：赛车在附着极限附近的侧偏、越野车在泥地和草地上的打滑，取决于路面，而路面每天都不一样。
软体与摩擦：气动软体手指、绳驱关节、减速器里的摩擦和间隙，物理建模要么太粗，要么太贵。

24.1 节讲了怎样从数据里学 $\mathbf x_{k+1}=f_\theta(\mathbf x_k,\mathbf u_k)$（那里按强化学习的习惯写作 $s,a$；本节沿用 13.4 节的控制记号 $\mathbf x,\mathbf u$），24.2 节和 24.3 节讲了用它学策略或在潜空间里规划的 model-based RL 方法。这一节站在控制工程师的角度：已经有一个 MPC，怎样把学到的模型接进去，让它仍然实时、仍然尊重约束，并且在模型不可信的地方不出事。

The MPC of Lesson 13.4 uses a model to predict the next few seconds at every cycle, optimizes under constraints, and applies the first step. It assumes the model is written down: the cart is a point mass, the bicycle model has a wheelbase, the quadruped is a single rigid body. Yet on many robots the dynamics that matter most are exactly the ones you cannot write down, or not accurately:

Aerodynamics: a quadrotor flying above 10 m/s suffers noticeable tracking errors from drag, rotor aerodynamic effects and the interaction of the rotor downwash with the body, which depend on speed, attitude and rotor speed and are hard to capture with a few parameters.
Tyres and terrain: the side-slip of a race car near the grip limit and the slip of an off-road vehicle on mud and grass depend on the surface, and the surface is different every day.
Soft bodies and friction: pneumatic soft fingers, tendon-driven joints, friction and backlash in gearboxes are either too crude or too expensive to model from physics.

Lesson 24.1 showed how to learn $\mathbf x_{k+1}=f_\theta(\mathbf x_k,\mathbf u_k)$ from data (written there as $s,a$ in the RL convention; this lesson keeps the control notation $\mathbf x,\mathbf u$ of Lesson 13.4), and Lessons 24.2 and 24.3 covered model-based RL methods that use such a model to learn a policy or to plan in a latent space. This lesson takes the control engineer's point of view: you already have an MPC; how do you plug a learned model into it so that it stays real-time, still respects constraints, and does not come to grief where the model cannot be trusted?

2直觉：规划器只看得见模型Intuition: the planner sees only the model

MPC 的优化器从不碰真实的机器人：它在模型里试遍各种动作序列，挑出模型认为最好的那一条。所以规划的好坏完全取决于模型在规划会去的那些状态上准不准。重新测量、重新求解（13.4 节的反馈）可以纠正一部分误差，但纠正不了“模型把整个方向都弄错了”的情形。

下图是一个力矩受限的单摆（$m=l=1$，$|u|\le5$ N·m，比重力矩 9.81 N·m 小，所以必须来回摆几次才能立起来），关节角 $q$ 从竖直向下量起，$q=\pm\pi$ 是倒立。真实动力学里还有黏性摩擦和库仑摩擦。每 0.05 s 用 MPPI（6.11 节）在 1.5 s 的时域上规划一次。四种模型可选：真实物理；“手册模型”（以为质量是 0.7 kg，并且没有摩擦）；以及两个从数据里学出来的网络 $\hl{fth}{f_\theta}$，输入 $(q,\dot q,u)$，输出这一步的角速度变化。两个网络结构完全一样（3→32→32→1，tanh），区别只在训练数据 $\hl{data}{\mathcal D}$：一个的 2000 个样本铺满整个相平面，另一个只来自摆在下方附近晃动（$|q|\le1.2$ rad，$|\dot q|\le3$ rad/s）。24.2 节的 PETS 在一个“学模型、规划、再采数据”的循环里让摆学会甩起；这里把循环拿掉，数据固定，只换 MPC 里的模型，单独看数据覆盖的作用。24.1 节的 FIG 3 已经看到，开环展开一旦离开数据就会崩；这里看的是闭环：每 0.05 s 重新测量、重新规划的反馈，能不能救回来。

MPC's optimizer never touches the real robot: it tries action sequences inside the model and picks the one the model likes best. So a plan is only as good as the model is on the states the plan visits. Measuring and re-solving (the feedback of Lesson 13.4) corrects some error, but not a model that gets a whole direction wrong.

The figure shows a torque-limited pendulum ($m=l=1$, $|u|\le5$ N·m, less than the 9.81 N·m of gravity, so it has to swing back and forth a few times before it can stand up). The joint angle $q$ is measured from hanging straight down, and $q=\pm\pi$ is upright. The true dynamics also contain viscous and Coulomb friction. Every 0.05 s, MPPI (Lesson 6.11) plans over a 1.5 s horizon. Four models are available: the true physics; a "handbook model" (believes the mass is 0.7 kg and that there is no friction); and two networks $\hl{fth}{f_\theta}$ learned from data, which take $(q,\dot q,u)$ and output this step's change in angular velocity. The two networks have exactly the same architecture (3→32→32→1, tanh) and differ only in their training data $\hl{data}{\mathcal D}$: one has 2000 samples spread over the whole phase plane, the other only comes from the pendulum swinging near the bottom ($|q|\le1.2$ rad, $|\dot q|\le3$ rad/s). The PETS of Lesson 24.2 learns the swing-up inside a "learn a model, plan, collect more data" loop; here the loop is removed, the data are fixed, and only the model inside the MPC changes, to isolate what data coverage does. FIG 3 of Lesson 24.1 already showed an open-loop rollout breaking down once it leaves the data; the question here is the closed loop: can the feedback of measuring and re-planning every 0.05 s rescue it?

FIG 1同一个 MPC，换一个模型：数据覆盖到哪里，规划就可靠到哪里Same MPC, different model: the plan is reliable only as far as the data reach

单摆PENDULUM

相平面 (q, q̇)PHASE PLANE (q, q̇)

相平面里绿色是真实走过的轨迹，蓝色虚线是此刻 MPC 用所选模型预测的 1.5 s，青色小点是所选网络的训练数据（选物理模型时不画）。琥珀色圆点是倒立目标 $q=\pm\pi$。左图的蓝色虚线是预测的摆尖轨迹。下方是两个网络的训练损失（对数坐标，蓝色 = 全相平面数据，紫色 = 只在下方的数据），损失按网络输出的归一化单位计。“竖起”指 $|q\mp\pi|<0.2$ rad 且 $|\dot q|<1$ rad/s。每次运行 8 s，选学到的模型前要先训练（约两三秒）。 In the phase plane, green is the trajectory actually travelled, the dashed blue curve is the 1.5 s the MPC is predicting right now with the chosen model, and the small cyan dots are the chosen network's training data (not drawn for the physics models). The amber dots mark the upright goal $q=\pm\pi$. On the left the dashed blue curve is the predicted path of the pendulum tip. Below are the training losses of the two networks (log scale; blue = whole-phase-plane data, purple = bottom-only data), in the network's normalized output units. "Upright" means $|q\mp\pi|<0.2$ rad and $|\dot q|<1$ rad/s. Each run lasts 8 s; the learned models must be trained first (two or three seconds).

试一试Try this

按“训练”，然后选“学到的 · 全相平面数据”并跑满 8 s：单摆在 4.5 s 内竖起，并且 8 s 里至少 3.5 s 处于竖起状态（这里是 4.00 s 时竖起、竖起 4.05 s，与真实物理的 4.05 s 和 4.00 s 几乎一样）。Press "Train", then choose "learned · whole-plane data" and run the full 8 s: the pendulum is upright within 4.5 s and spends at least 3.5 s of the 8 s upright (here it gets up at 4.00 s and stays up 4.05 s, almost the same as the true physics' 4.05 s and 4.00 s).
再选“学到的 · 只在下方的数据”跑满 8 s：它一次也没有竖起来。可读数显示，在它有数据的区域里，它比全数据网络还准（加速度误差 0.15 对 0.58 rad/s²）；问题出在整个相平面上（4.52 对 0.51 rad/s²）。看蓝色预测在摆过水平之后怎样和绿色分道扬镳。Now choose "learned · bottom-only data" and run the full 8 s: it never gets upright. Yet the readout shows that inside the region where it has data it is even more accurate than the whole-plane network (acceleration error 0.15 versus 0.58 rad/s²); the problem is the phase plane as a whole (4.52 versus 0.51 rad/s²). Watch the blue prediction part ways with the green path once the pendulum swings past horizontal.

核心洞见Key insight

对 MPC 来说，“模型准不准”不是一个数，而是一张地图：在规划会去的那些状态上准才算准。手册模型质量错了 30%、还没有摩擦，可它的误差是“全局温和”的，反馈能吸收，所以 4.50 s 就竖起来了；只见过下方数据的网络在训练分布里更准，可规划器偏偏要把摆带到它从没见过的地方，在那里它说的话毫无根据。这正是后面几节的主线：要么让数据覆盖规划会去的地方，要么让规划器知道哪里不能信。

For MPC, "how accurate is the model" is not one number but a map: what counts is accuracy on the states the plan will visit. The handbook model has a 30% mass error and no friction, but its error is "mild everywhere", feedback absorbs it, and the pendulum is upright at 4.50 s. The bottom-only network is more accurate inside its training distribution, but the planner insists on taking the pendulum where it has never seen data, and there it is making things up. That is the thread of the rest of this lesson: either let the data cover where the plan goes, or let the planner know where not to trust the model.

3动手玩：在学到的模型上，用梯度还是用采样？Play: gradients or sampling on a learned model?

神经网络模型有一个物理仿真器常常没有的好处：它处处可导。16.3 节的反向传播不但能求损失对权重的梯度，也能求网络输出对输入的梯度；把 $N$ 步展开串起来，再沿时间往回传（与 6.13 节的打靶法相同，只是 Jacobian 由网络给出），就得到预测代价对整串动作的梯度 $\hl{grad}{\nabla_{\mathbf U}J}$。另一条路是 6.11 节和 13.4 节的 MPPI：把网络当黑盒，撒一批动作序列，按代价加权平均（权重 $\hl{wts}{w_i}$）。

下图两个单摆用同一个学到的网络（FIG 1 的全数据网络）和同样的代价，每个控制周期给同样多的网络计算预算（一次前向记 1 份，一次反向记 2 份）。蓝色用梯度：每周期从平移后的上一计划出发做几步投影 Adam；琥珀色用 MPPI。打开“禁区”后，代价里加一项：摆的角度落在 $q\in(-2.6,-1.6)$ rad 这块扇形里，每一步罚 50。它是一个指示函数，几乎处处梯度为 0。

A neural-network model has an advantage that a physics simulator often lacks: it is differentiable everywhere. The backpropagation of Lesson 16.3 gives not only the gradient of a loss with respect to the weights but also the gradient of the network's output with respect to its inputs. Chain $N$ steps of the rollout together and propagate backwards through time (the shooting method of Lesson 6.13, with the Jacobians supplied by the network), and you get the gradient of the predicted cost with respect to the whole action sequence, $\hl{grad}{\nabla_{\mathbf U}J}$. The other road is the MPPI of Lessons 6.11 and 13.4: treat the network as a black box, scatter a batch of action sequences and take a cost-weighted average (weights $\hl{wts}{w_i}$).

In the figure two pendulums use the same learned network (FIG 1's whole-plane network) and the same cost, and get the same network budget per control cycle (a forward pass counts 1, a backward pass 2). Blue uses gradients: each cycle it takes a few projected Adam steps starting from the shifted previous plan; amber uses MPPI. Switching on the "no-go zone" adds a term to the cost: 50 for every step the pendulum angle spends in the wedge $q\in(-2.6,-1.6)$ rad. That is an indicator function, whose gradient is zero almost everywhere.

FIG 2同样的预算：穿过网络求梯度 vs. 在网络里采样Equal budget: gradients through the network vs. sampling inside it

上：左边蓝色单摆用梯度规划，右边琥珀色单摆用 MPPI；红色扇形是禁区（关闭时画成虚线轮廓）。下：两者的“离倒立的高度” $1+\cos q$（0 = 倒立，2 = 竖直向下）随时间变化。预算 300 时，梯度法每周期做 3 步（每步一次 30 步展开的前向和反向），MPPI 用 10 条样本；预算 900 时分别是 10 步和 30 条；1800 时是 20 步和 60 条。“ms/周期”是你的浏览器里实测的计算时间。 Top: the blue pendulum on the left plans with gradients, the amber one on the right with MPPI; the red wedge is the no-go zone (drawn as a dashed outline when it is off). Bottom: each one's "height below upright" $1+\cos q$ (0 = upright, 2 = hanging down) over time. At budget 300 the gradient method takes 3 steps per cycle (each a forward and backward pass through a 30-step rollout) and MPPI uses 10 samples; at 900 it is 10 steps and 30 samples; at 1800, 20 steps and 60 samples. "ms/cycle" is the computation time measured in your browser.

试一试Try this

禁区关闭，预算 300，跑满 8 s：梯度法比 MPPI 先竖起来（3.40 s 对 5.35 s）。把预算加到 1800，MPPI 追近了（4.05 s），梯度法几乎没变（3.45 s）。With the zone off and budget 300, run the full 8 s: the gradient planner is upright before MPPI (3.40 s versus 5.35 s). Raise the budget to 1800 and MPPI closes in (4.05 s) while the gradient planner barely changes (3.45 s).
打开禁区，预算 900 或 1800，跑满 8 s：梯度法照样穿过禁区（900 时 13 个周期落在里面），MPPI 一次也没进去，代价是晚一点竖起来（5.30 s）。Switch the zone on, use budget 900 or 1800, and run the full 8 s: the gradient planner goes straight through the zone (13 cycles inside it at 900), while MPPI never enters it, at the price of getting upright later (5.30 s).

核心洞见Key insight

可导的模型让梯度法在代价光滑时用很少的计算就找到好计划：一次反向传播给出所有 $N$ 个动作的方向，而采样要靠很多条样本“摸”出同样的信息。可梯度只看得见局部的斜率：指示函数、碰撞、接触切换这类不连续的代价，它根本感觉不到（预算 300 时梯度法没进禁区，只是因为它恰好往另一侧摆）。采样不需要导数，能处理任何代价，代价是在高维动作空间里需要更多样本。实践中两者常常合用：TD-MPC（24.3 节）用 MPPI 在学到的潜模型里规划，也有工作先采样找到好的盆地，再用梯度精修。

A differentiable model lets gradients find a good plan with very little computation when the cost is smooth: one backward pass yields a direction for all $N$ actions, whereas sampling needs many samples to "feel out" the same information. But a gradient sees only the local slope: discontinuous costs such as indicators, collisions and contact switches are invisible to it (at budget 300 the gradient planner avoided the zone only because it happened to swing the other way). Sampling needs no derivatives and handles any cost, at the price of needing more samples in high-dimensional action spaces. In practice the two are often combined: TD-MPC (Lesson 24.3) plans with MPPI inside a learned latent model, and other work samples first to find a good basin and then refines with gradients.

4第二个视角：把非线性模型“抬”成线性的A second view: lift a nonlinear model until it is linear

FIG 2 的两个规划器都在解一个非凸问题，没有收敛保证，也难以加硬约束。13.4 节告诉我们，线性模型 + 二次代价 + 线性约束是一个凸 QP，几毫秒就能解到全局最优。能不能让学到的模型是线性的，同时又能描述非线性系统？

Koopman 算子理论给了一个答案：不直接预测状态 $\mathbf x$，而是预测状态的一组函数 $\hl{psi}{\boldsymbol\psi}(\mathbf x)$（“观测量”或“提升函数”）。对很多非线性系统，在足够丰富的函数空间里，这些函数的演化近似是线性的：$\boldsymbol\psi(\mathbf x_{k+1})\approx A\boldsymbol\psi(\mathbf x_k)+B\mathbf u_k$。$A,B$ 用最小二乘（1.13 节）从数据里一次算出，这叫扩展动态模态分解（EDMD）。因为 $\mathbf u$ 仍然线性地进入、状态又是 $\boldsymbol\psi$ 的前两个分量，MPC 在提升空间里就是一个标准的线性 MPC，每个周期一个凸 QP；缩并之后，QP 的规模只取决于时域和输入维数，与提升维数基本无关（Korda 与 Mezić，2018）。

下图的系统是一个刚度随形变变硬的弹簧（气动软体执行器常有这种特性）：$\ddot x=-x-2x^3-0.5\dot x+u$，$|u|\le12$。小位移时它像线性弹簧，大位移时 $x^3$ 项占主导。四种模型对比：在原点处的局部线性化 $\hl{Alin}{A_{\rm lin}}$（紫色，“在工作点附近把模型线性化”的经典做法）；以及从 4000 个随机激励的数据点用 EDMD 学出的模型（蓝色），提升函数可选：不提升（只用 $x,\dot x$，即普通的线性回归，也叫 DMD）、多项式 $[x,\dot x,x^3,x^2\dot x,x^5]$、10 个或 30 个径向基函数。

Both planners of FIG 2 solve a nonconvex problem with no convergence guarantee, and hard constraints are awkward for them. Lesson 13.4 showed that a linear model with a quadratic cost and linear constraints is a convex QP, solved to the global optimum in milliseconds. Can the learned model be linear and still describe a nonlinear system?

Koopman operator theory offers an answer: instead of predicting the state $\mathbf x$ directly, predict a set of functions of the state $\hl{psi}{\boldsymbol\psi}(\mathbf x)$ ("observables" or "lifting functions"). For many nonlinear systems, in a rich enough space of functions, these functions evolve approximately linearly: $\boldsymbol\psi(\mathbf x_{k+1})\approx A\boldsymbol\psi(\mathbf x_k)+B\mathbf u_k$. $A$ and $B$ come out of the data in one least-squares solve (Lesson 1.13); this is extended dynamic mode decomposition (EDMD). Because $\mathbf u$ still enters linearly and the state is the first two components of $\boldsymbol\psi$, MPC in the lifted space is a standard linear MPC, one convex QP per cycle; after condensing, the QP's size depends only on the horizon and the input dimension, hardly on the lifted dimension (Korda & Mezić, 2018).

The system below is a spring that stiffens as it deforms (a common trait of pneumatic soft actuators): $\ddot x=-x-2x^3-0.5\dot x+u$, $|u|\le12$. For small deflections it behaves like a linear spring; for large ones the $x^3$ term dominates. Four kinds of model are compared: the local linearization at the origin $\hl{Alin}{A_{\rm lin}}$ (purple; the classic "linearize the model around the operating point"), and models learned with EDMD from 4000 randomly excited data points (blue), with a choice of lifting: no lift (only $x,\dot x$, i.e. ordinary linear regression, also called DMD), the polynomials $[x,\dot x,x^3,x^2\dot x,x^5]$, or 10 or 30 radial basis functions.

FIG 3Koopman 提升：线性的模型，非线性的预测，凸的 MPCKoopman lifting: a linear model, nonlinear predictions, a convex MPC

自由响应预测：拖动琥珀色初值FREE-RESPONSE PREDICTION · drag the amber start

闭环跟踪 x(t)CLOSED-LOOP TRACKING x(t)

左：从琥珀色初值出发、$u=0$ 的 2 s 自由响应，绿色是真实系统，紫色是局部线性化的预测，蓝色是所选 EDMD 模型的预测（每一步都在提升空间里线性推进，从不回到 $\mathbf x$ 再重新提升）。右：两个线性 MPC（时域 20 步 = 1 s，代价 $\sum_k(x_k-r_k)^2+10^{-3}u_k^2$，$|u|\le12$，每周期一个 20 变量的盒约束 QP）跟踪幅值为 $A$、频率 0.25 Hz 的正弦参考（琥珀色虚线）。每个周期都从测到的真实状态重新提升 $\boldsymbol\psi(\mathbf x)$。读数里的“20 步预测误差”是在 200 段随机激励的测试轨迹上，1 s 后位置误差的平均值。 Left: the 2 s free response ($u=0$) from the amber start; green is the true system, purple the local linearization's prediction, blue the chosen EDMD model's prediction (advanced linearly in the lifted space at every step, never mapped back to $\mathbf x$ and re-lifted). Right: two linear MPCs (horizon 20 steps = 1 s, cost $\sum_k(x_k-r_k)^2+10^{-3}u_k^2$, $|u|\le12$, one 20-variable box-constrained QP per cycle) track a sine reference of amplitude $A$ at 0.25 Hz (dashed amber). Every cycle re-lifts $\boldsymbol\psi(\mathbf x)$ from the measured true state. The "20-step prediction error" in the readout is the mean position error after 1 s over 200 randomly excited test trajectories.

试一试Try this

选“不提升”，先把幅值调到 $A\le0.5$：局部线性化跟踪得比数据拟合的线性模型好（$A=0.5$ 时平均误差 0.008 对 0.042）；再调到 $A\ge1.5$：反过来了（$A=1.5$ 时 0.097 对 0.041）。两个都是线性模型，各自只在一个范围里好。Choose "no lift" and first set the amplitude to $A\le0.5$: the local linearization tracks better than the linear model fitted to data (mean error 0.008 versus 0.042 at $A=0.5$); then set $A\ge1.5$: it is the other way round (0.097 versus 0.041 at $A=1.5$). Both are linear models, and each is good only over one range.
把幅值调到 $A\ge1.5$，换一种提升函数，让蓝色的跟踪误差不到紫色的三分之一（例如 $A=1.8$ 时多项式提升 0.040 对 0.149）。再把幅值调回 0.3：多项式提升在小幅值时也和局部线性化一样好（0.004 对 0.004；10 个 RBF 太粗，这里是 0.017）。Set the amplitude to $A\ge1.5$ and choose a lifting whose blue tracking error is less than a third of purple's (for instance polynomial lifting at $A=1.8$: 0.040 versus 0.149). Then set the amplitude back to 0.3: the polynomial lift is as good as the local linearization at small amplitude too (0.004 versus 0.004; 10 RBFs are too coarse and give 0.017 here).

核心洞见Key insight

提升让“线性”成为模型的形式，而不是对系统的假设：$A,B$ 仍是线性的，所以 MPC 仍是凸 QP；非线性藏在 $\boldsymbol\psi$ 里。这不是免费的。有限个提升函数只能近似 Koopman 算子，预测会随时域变差：20 步时四种提升的误差是 0.24、0.21、0.17、0.16，40 步时都在 0.43 到 0.45 之间。好的提升函数取决于系统（这里多项式里恰好有 $x^3$），而控制输入线性进入只是一个假设；对于输入和状态强耦合的系统（例如力矩乘以 $\cos q$），这种形式本身就有误差。

Lifting makes "linear" a property of the model's form rather than an assumption about the system: $A,B$ are still linear, so MPC is still a convex QP, and the nonlinearity hides in $\boldsymbol\psi$. It is not free. Finitely many lifting functions only approximate the Koopman operator, and predictions degrade with the horizon: at 20 steps the four lifts have errors of 0.24, 0.21, 0.17 and 0.16, while at 40 steps they are all between 0.43 and 0.45. Good lifting functions depend on the system (here the polynomials happen to contain $x^3$), and letting the input enter linearly is an assumption; for systems where input and state are strongly coupled (a torque multiplied by $\cos q$, say), this form is wrong in itself.

5第三个视角：让规划器知道模型在哪里没把握A third view: tell the planner where the model is unsure

代价里的不确定性：悲观、中性还是乐观Uncertainty in the cost: pessimistic, neutral or optimistic

FIG 1 的失败有一个更深的原因，24.2 节的直觉部分已经看到：优化器在主动寻找模型说得好听的地方。模型误差在某处碰巧偏乐观，规划就会往那里去。24.2 节用集成的分歧给候选动作悲观地打分；这里把同一个想法放进 MPC 的代价和约束里。21.3 节的认知不确定性正好标出了这些地方：没有数据的地方，模型只是在猜。

下图是一辆巡视车要从左边穿过一片地到右边。地里藏着泥（真实的泥塘画成红色虚线圆，车不知道），泥里的速度只有平地的 15%。车用一个高斯过程（GP，第 6 节）学“地面速度系数” $s(\mathbf p)$，先验是 $s=1$（没去过的地方默认是平地）。它只走过一次下方的那条绕行路线（绿色数据点）。规划时把速度换成一个“置信下界”：

The failure in FIG 1 has a deeper cause, already seen in the intuition of Lesson 24.2: the optimizer is actively searching for places where the model says nice things. Wherever the model's error happens to be optimistic, the plan goes there. Lesson 24.2 scored candidate actions pessimistically with the ensemble's disagreement; here the same idea goes into MPC's cost and constraints. The epistemic uncertainty of Lesson 21.3 marks exactly these places: where there are no data, the model is guessing.

In the figure a rover must cross a field from left to right. Mud is hidden in the field (the true mud pits are drawn as dashed red circles; the rover does not know about them), and in the mud the rover moves at only 15% of its speed on firm ground. The rover learns a "ground speed factor" $s(\mathbf p)$ with a Gaussian process (GP, Section 6) whose prior is $s=1$ (unvisited ground is assumed firm). It has driven only once, along the detour at the bottom (the green data points). For planning, the speed is replaced by a "confidence bound":

$$ \mathbf p_{k+1}=\mathbf p_k+\Delta t\;\max\!\big(0.1,\;\mu_s(\mathbf p_k)-\hl{beta}{\beta}\,\hl{sig}{\sigma_s(\mathbf p_k)}\big)\begin{bmatrix}\cos a_k\\ \sin a_k\end{bmatrix},\qquad J=\sum_{k=1}^{N}\|\mathbf p_k-\mathbf p_{\rm goal}\| $$

$\beta>0$ 是悲观（“没把握的地方按最坏的情况算”），$\beta=0$ 只用均值，$\beta<0$ 是乐观（“没把握的地方可能很好，去看看”，即探索）。MPPI 每 0.25 s 对 24 步的航向序列规划一次。每跑完一趟，车把沿途每 0.3 m 测到的速度加进 GP 的数据里。

$\beta>0$ is pessimistic ("where unsure, assume the worst"), $\beta=0$ uses the mean only, and $\beta<0$ is optimistic ("where unsure it might be good; go and look", i.e. exploration). Every 0.25 s MPPI plans a 24-step heading sequence. After each run the rover adds the speeds it measured every 0.3 m along its path to the GP's data.

FIG 4没去过的地方：按均值规划会陷进去，按置信下界规划会绕开Unvisited ground: planning on the mean drives in, planning on a lower bound drives around

底色是 GP 的估计：越红 = 估计的速度系数 $\mu_s$ 越低，越青 = 不确定性 $\sigma_s$ 越大（没去过的地方）。绿色小点是数据，琥珀色实线是这一趟实际走的路线，蓝色虚线是此刻的计划。“出发时预测”是出发那一刻规划器（用它的模型和 $\beta$）认为到达终点要多久，“实际”是真实用时。动画按 4 倍速播放。 The shading is the GP's estimate: more red = lower estimated speed factor $\mu_s$, more cyan = larger uncertainty $\sigma_s$ (unvisited ground). The small green dots are the data, the solid amber line is this run's actual path, and the dashed blue line is the current plan. "Predicted at start" is how long the planner (with its model and $\beta$) believed the trip would take when it set off; "actual" is the real time. The animation plays at 4× speed.

试一试Try this

先按“重置数据”，用 $\beta=0$ 跑第一趟：车直奔终点，陷进泥里，实际用时是出发时预测的两倍以上（21.5 s 对 8.55 s）。接着不重置再跑两趟：第三趟学乖了，绕开泥塘，9.0 s 到达。Press "Reset data" and make the first run with $\beta=0$: the rover heads straight for the goal, gets stuck in the mud, and takes more than twice the time it predicted at the start (21.5 s versus 8.55 s). Then make two more runs without resetting: by the third run it has learned, drives around the mud and arrives in 9.0 s.
按“重置数据”，把 $\beta$ 调到 1.5 或更大再跑第一趟：第一次就没有一步踩进泥里（$\beta=2$ 时 10.0 s）。再试 $\beta=1$：它仍然陷进去（18.75 s）。为什么“有点悲观”还不够？（提示：24 步 × 0.25 s 的时域看得有多远？）Press "Reset data", set $\beta$ to 1.5 or more and make the first run: not a single step lands in the mud the very first time (10.0 s at $\beta=2$). Now try $\beta=1$: it still gets stuck (18.75 s). Why is "a bit pessimistic" not enough? (Hint: how far does a 24-step × 0.25 s horizon see?)

约束里的不确定性：高斯过程 MPC 与机会约束Uncertainty in the constraints: Gaussian-process MPC and chance constraints

代价里的 $\beta$ 改变“想去哪”，约束里的不确定性决定“离边界留多少余量”。Hewing 等人的谨慎 MPC（cautious MPC）是这条路线的代表：标称模型加一个 GP 学到的残差 $\mathbf x_{k+1}=f(\mathbf x_k,\mathbf u_k)+B_d\,d(\mathbf x_k)$，$d\sim\mathcal{GP}$；沿预测时域同时传播均值和方差，把状态约束写成机会约束 $\Pr(h(\mathbf x_k)\le0)\ge1-\varepsilon$，再近似成收紧的确定性约束 $h(\boldsymbol\mu_k)+\hl{tube}{\kappa\,\sigma_{h,k}}\le0$。

下图：小车（加速度输入 $|u|\le1$ m/s²，与 13.4 节同款）从 $p=-4$ 出发，要停在离墙 25 cm 的 $p=-0.25$。墙前最后一米是一段下坡，车不知道，坡给它额外 0.4 m/s² 的推力，刹车只剩六成力气。GP 学这个残差加速度 $d(p)$，它只在 $p\in[-4.2,-1.8]$ 有数据（以前来过的地方）。“标称”模式不用 GP；“GP 均值”把 $\hl{gpmean}{\mu_d}$ 放进预测；“GP + 收紧”还要求预测位置加 $\kappa$ 倍标准差不越过墙。CEM 每 0.1 s 在 2.5 s 时域上规划一次；每跑完一趟，沿途的残差测量加进数据。

$\beta$ in the cost changes "where to go"; uncertainty in the constraints decides "how much margin to keep from a boundary". The cautious MPC of Hewing et al. is the standard example: a nominal model plus a residual learned by a GP, $\mathbf x_{k+1}=f(\mathbf x_k,\mathbf u_k)+B_d\,d(\mathbf x_k)$ with $d\sim\mathcal{GP}$; propagate the mean and the variance along the horizon together, write the state constraints as chance constraints $\Pr(h(\mathbf x_k)\le0)\ge1-\varepsilon$, and approximate them by tightened deterministic constraints $h(\boldsymbol\mu_k)+\hl{tube}{\kappa\,\sigma_{h,k}}\le0$.

The figure: a cart (acceleration input $|u|\le1$ m/s², the same as in Lesson 13.4) leaves $p=-4$ and must stop at $p=-0.25$, 25 cm from a wall. The last metre before the wall is a downhill ramp the cart does not know about; it adds 0.4 m/s² of push, so braking has only 60% of its strength left. A GP learns this residual acceleration $d(p)$, with data only in $p\in[-4.2,-1.8]$ (where the cart has been before). "Nominal" mode uses no GP; "GP mean" puts $\hl{gpmean}{\mu_d}$ into the prediction; "GP + tightening" also requires the predicted position plus $\kappa$ standard deviations to stay off the wall. CEM plans over a 2.5 s horizon every 0.1 s; after each run, the residual measurements along the way are added to the data.

FIG 5GP-MPC：不确定的地方，约束自动收紧GP-MPC: where the model is unsure, the constraint tightens itself

残差加速度 d(p)（m/s²）RESIDUAL ACCELERATION d(p) (m/s²)

位置 p(t)（m）与此刻的预测管道POSITION p(t) (m) AND THE CURRENT PREDICTED TUBE

上：绿色虚线是真实的残差（坡），蓝色是 GP 均值，青色带是 ±2 个标准差，绿色小点是数据，琥珀色竖线是小车此刻的位置。下：琥珀色实线是这一趟的 $p(t)$，蓝色是此刻的均值预测，青色带是 $\pm\kappa\sigma_p$ 的预测管道（$\kappa=0$ 时画成 $\pm\sigma_p$ 作参考），红线是墙 $p=0$，绿色虚线是目标。预测管道的宽度来自 GP 的标准差沿时域的传播（第 6 节）。 Top: the dashed green curve is the true residual (the ramp), blue is the GP mean, the cyan band is ±2 standard deviations, the small green dots are the data, and the amber vertical line is the cart's current position. Bottom: the solid amber curve is this run's $p(t)$, blue is the current mean prediction, the cyan band is the predicted tube $\pm\kappa\sigma_p$ (drawn as $\pm\sigma_p$ for reference when $\kappa=0$), the red line is the wall $p=0$ and the dashed green line the target. The tube's width comes from propagating the GP's standard deviation along the horizon (Section 6).

试一试Try this

按“重置数据”，用“标称”或“GP 均值”跑第一趟：小车撞上墙（越过 $p=0$，标称时最远到 $+0.26$ m）。坡附近没有数据，GP 均值就等于先验 0，和标称模型一样乐观。Press "Reset data" and make a first run with "nominal" or "GP mean": the cart hits the wall (goes past $p=0$; up to $+0.26$ m with the nominal model). There are no data near the ramp, so the GP mean equals its prior of 0 and is as optimistic as the nominal model.
按“重置数据”，选“GP + 收紧”，$\kappa\ge2$，跑第一趟：小车停在墙前（$\kappa=2$ 时离墙 2.4 cm，7.1 s 才到目标）。不重置再跑一趟：数据有了，管道变窄，4.6 s 以内就停到目标，离墙仍有 20 cm 以上。Press "Reset data", choose "GP + tightening" with $\kappa\ge2$, and make the first run: the cart stops short of the wall (2.4 cm from it at $\kappa=2$, reaching the target only at 7.1 s). Run again without resetting: now there are data, the tube is narrower, and the cart settles at the target within 4.6 s, still more than 20 cm from the wall.

核心洞见Key insight

“不知道”本身是可以用的信息。认知不确定性只在没数据的地方大，所以不确定性感知的 MPC 在熟悉的地方和普通 MPC 一样快，在陌生的地方自动变谨慎，数据一多，余量就自己收回来（FIG 5 第二趟）。代价是它依赖不确定性估计本身靠得住：GP 的核长度、先验均值选错，$\sigma$ 就会说错（FIG 4 若先验是“泥地”，车就永远不敢走直线）；而 $\kappa$、$\beta$ 的大小与“出错的概率”之间的对应，只在模型假设成立时才精确。

"Not knowing" is itself usable information. Epistemic uncertainty is large only where there are no data, so uncertainty-aware MPC is as fast as ordinary MPC in familiar territory and automatically cautious in unfamiliar territory, and once data arrive the margin shrinks back by itself (FIG 5's second run). The price is that it relies on the uncertainty estimate being trustworthy: choose the GP's length scale or prior mean badly and $\sigma$ will be wrong (if FIG 4's prior were "mud", the rover would never dare drive straight); and the link between the size of $\kappa$ or $\beta$ and "the probability of failure" is exact only if the model's assumptions hold.

6再看数学The math

提示：把鼠标悬停在公式中带底色的符号上，对应的图元素会被高亮（FIG 1 的预测曲线或数据点、FIG 2 的蓝色或琥珀色单摆、FIG 3 的蓝色或紫色曲线、FIG 4 的不确定性底色或计划路线（跑动时）、FIG 5 的 GP 均值或预测管道（跑动时）、FIG 6 的紫色残差箭头（有数据之后））。Tip: hover the highlighted symbols in the formulas; the matching element lights up (FIG 1's predicted curve or data points, FIG 2's blue or amber pendulum, FIG 3's blue or purple curves, FIG 4's uncertainty shading or, during a run, its planned route; FIG 5's GP mean or, during a run, its predicted tube; FIG 6's purple residual arrows, once there are data).

学到的模型进入 MPCThe learned model inside MPC

$$ \min_{\mathbf u_0,\dots,\mathbf u_{N-1}}\ \sum_{k=0}^{N-1}\ell(\mathbf x_{k+1},\mathbf u_k)\quad\text{s.t.}\quad \mathbf x_{k+1}=\hl{fth}{f_\theta}(\mathbf x_k,\mathbf u_k),\ \ \mathbf x_0=\mathbf x(t),\ \ \mathbf u_k\in\mathcal U $$

与 13.4 节唯一的不同是 $f_\theta$ 由数据 $\hl{data}{\mathcal D}=\{(\mathbf x_i,\mathbf u_i,\mathbf x_i^+)\}$ 拟合而来（24.1 节）。FIG 1 的网络预测角速度增量，位置用半隐式欧拉积分：$\dot q_{k+1}=\dot q_k+\Delta t\,\hat a_\theta(q_k,\dot q_k,u_k)$，$q_{k+1}=q_k+\Delta t\,\dot q_{k+1}$。这种“结构已知的部分写死，只学未知的部分”的做法本身就是一种先验（24.1 节的“预测增量”）。

The only difference from Lesson 13.4 is that $f_\theta$ is fitted to data $\hl{data}{\mathcal D}=\{(\mathbf x_i,\mathbf u_i,\mathbf x_i^+)\}$ (Lesson 24.1). FIG 1's network predicts the increment of angular velocity, and the angle is integrated by semi-implicit Euler: $\dot q_{k+1}=\dot q_k+\Delta t\,\hat a_\theta(q_k,\dot q_k,u_k)$, $q_{k+1}=q_k+\Delta t\,\dot q_{k+1}$. Hard-coding the part whose structure is known and learning only the unknown part is itself a prior (the "predict increments" of Lesson 24.1).

梯度：沿时间反向传播Gradients: backpropagation through time

令 $J=\sum_k\ell(\mathbf x_{k+1},\mathbf u_k)$，并把含 $\mathbf x_k$ 的那一项记作 $\ell_k=\ell(\mathbf x_k,\mathbf u_{k-1})$。从最后一步往回递推伴随变量 $\boldsymbol\lambda_k=\partial J/\partial\mathbf x_k$：

Let $J=\sum_k\ell(\mathbf x_{k+1},\mathbf u_k)$, and write $\ell_k=\ell(\mathbf x_k,\mathbf u_{k-1})$ for the term that contains $\mathbf x_k$. Recurse the adjoint $\boldsymbol\lambda_k=\partial J/\partial\mathbf x_k$ backwards from the last step:

$$ \boldsymbol\lambda_N=\nabla_{\mathbf x}\ell_N,\qquad \boldsymbol\lambda_k=\nabla_{\mathbf x}\ell_k+\Big(\frac{\partial f_\theta}{\partial\mathbf x}\Big)^{\!\top}_{\!k}\boldsymbol\lambda_{k+1},\qquad \hl{grad}{\frac{\partial J}{\partial\mathbf u_k}}=\nabla_{\mathbf u}\ell_{k+1}+\Big(\frac{\partial f_\theta}{\partial\mathbf u}\Big)^{\!\top}_{\!k}\boldsymbol\lambda_{k+1} $$

每一个 $(\partial f_\theta/\partial\cdot)^\top\boldsymbol\lambda$ 正是对网络做一次反向传播、把上游梯度设为 $\boldsymbol\lambda_{k+1}$ 时得到的“对输入的梯度”，不需要显式构造 Jacobian。所以一次梯度的代价约等于一次前向展开加一次反向展开（FIG 2 记为 3 份）。动作约束 $|u|\le u_{\max}$ 用投影处理：每步更新后截断到盒子里。

Each $(\partial f_\theta/\partial\cdot)^\top\boldsymbol\lambda$ is exactly the "gradient with respect to the input" that one backward pass through the network returns when the upstream gradient is set to $\boldsymbol\lambda_{k+1}$; no Jacobian is ever formed. One gradient therefore costs about one forward rollout plus one backward rollout (counted as 3 units in FIG 2). The action limit $|u|\le u_{\max}$ is handled by projection: clip into the box after every update.

采样：MPPI 与 CEMSampling: MPPI and CEM

MPPI 在当前计划 $\mathbf U$ 周围采 $K$ 条 $\mathbf U+\boldsymbol\epsilon_i$，在模型里展开得到 $J_i$，然后 $\mathbf U\leftarrow\sum_i\hl{wts}{w_i}(\mathbf U+\boldsymbol\epsilon_i)$，$w_i\propto\exp(-(J_i-J_{\min})/\lambda)$；CEM 只平均最好的若干条，并更新采样方差（6.11 节）。FIG 2 的 MPPI 用 $\sigma=3$ N·m、$\lambda=0.3$；FIG 5 用 CEM（48 条、精英 6 条、每周期 3 轮）。它们只需要模型的前向计算，所以模型可以是网络、GP、甚至不可导的仿真器。

MPPI samples $K$ sequences $\mathbf U+\boldsymbol\epsilon_i$ around the current plan $\mathbf U$, rolls them out in the model to get $J_i$, and sets $\mathbf U\leftarrow\sum_i\hl{wts}{w_i}(\mathbf U+\boldsymbol\epsilon_i)$ with $w_i\propto\exp(-(J_i-J_{\min})/\lambda)$; CEM averages only the best few and updates the sampling variance (Lesson 6.11). FIG 2's MPPI uses $\sigma=3$ N·m and $\lambda=0.3$; FIG 5 uses CEM (48 samples, 6 elites, 3 rounds per cycle). They need only forward evaluations of the model, so the model can be a network, a GP, or even a non-differentiable simulator.

优化器在利用模型误差The optimizer exploits model error

24.1 节称之为“模型被利用”，24.2 节给出了直觉；这里写成式子。写 $\hat J(\mathbf U)=J(\mathbf U)+\varepsilon(\mathbf U)$，$\varepsilon$ 是模型误差造成的代价误差。规划器求的是 $\arg\min_{\mathbf U}\hat J$，它会偏向 $\varepsilon$ 为负（模型过于乐观）的那些 $\mathbf U$。即使对每个固定的 $\mathbf U$ 都有 $\mathbb E[\varepsilon]=0$，也有 $\mathbb E[\min_{\mathbf U}\hat J]\le\min_{\mathbf U}J$：优化后的预测代价系统性地偏低（“优化者的诅咒”）。$\varepsilon$ 的散布在没有数据的地方最大，所以这种偏差专门把计划拉向没数据的地方。这就是 FIG 1 窄数据网络、FIG 4 $\beta=0$ 时出事的机制，也是用不确定性修正代价的理由。

Lesson 24.1 called this "model exploitation" and Lesson 24.2 gave the intuition; here it is as a formula. Write $\hat J(\mathbf U)=J(\mathbf U)+\varepsilon(\mathbf U)$, where $\varepsilon$ is the cost error caused by model error. The planner computes $\arg\min_{\mathbf U}\hat J$, so it is drawn towards the $\mathbf U$ where $\varepsilon$ is negative (where the model is too optimistic). Even if $\mathbb E[\varepsilon]=0$ for every fixed $\mathbf U$, we get $\mathbb E[\min_{\mathbf U}\hat J]\le\min_{\mathbf U}J$: the optimized predicted cost is systematically too low (the "optimizer's curse"). The spread of $\varepsilon$ is largest where there are no data, so this bias specifically pulls plans towards data-free regions. That is the mechanism behind the narrow-data network of FIG 1 and FIG 4 at $\beta=0$, and the reason for correcting the cost with uncertainty.

Koopman 提升与 EDMDKoopman lifting and EDMD

对自治系统 $\mathbf x^+=F(\mathbf x)$，Koopman 算子作用在函数上：$(\mathcal K g)(\mathbf x)=g(F(\mathbf x))$。它是线性的（$\mathcal K(ag+bh)=a\mathcal Kg+b\mathcal Kh$），代价是它作用在无穷维的函数空间上。EDMD 取有限个函数 $\hl{psi}{\boldsymbol\psi}=[\psi_1,\dots,\psi_n]^\top$，用数据求它在这个子空间上的最小二乘近似；Korda 与 Mezić 把输入写成线性项：

For an autonomous system $\mathbf x^+=F(\mathbf x)$, the Koopman operator acts on functions: $(\mathcal K g)(\mathbf x)=g(F(\mathbf x))$. It is linear ($\mathcal K(ag+bh)=a\mathcal Kg+b\mathcal Kh$), at the price of acting on an infinite-dimensional space of functions. EDMD takes finitely many functions $\hl{psi}{\boldsymbol\psi}=[\psi_1,\dots,\psi_n]^\top$ and finds its least-squares approximation on that subspace from data; Korda and Mezić add the input as a linear term:

$$ \min_{A,B}\ \sum_i\big\|\boldsymbol\psi(\mathbf x_i^+)-A\boldsymbol\psi(\mathbf x_i)-B\mathbf u_i\big\|^2,\qquad \mathbf y_{k+1}=A\mathbf y_k+B\mathbf u_k,\quad \mathbf y_0=\boldsymbol\psi(\mathbf x(t)),\quad \hat{\mathbf x}_k=C\mathbf y_k $$

这里 $\boldsymbol\psi$ 的前两个分量就是 $x,\dot x$，所以 $C=[I\ 0]$。闭式解是正规方程 $[A\ B]^\top=(Z^\top Z)^{-1}Z^\top Y^+$，$Z$ 的每行是 $[\boldsymbol\psi(\mathbf x_i)^\top,\mathbf u_i^\top]$。有了线性模型，13.4 节的缩并就照搬：$\hat x_{k}=\mathbf c^\top A^k\mathbf y_0+\sum_{j\lt k}\mathbf c^\top A^{k-1-j}B\,u_j$，代价是 $U$ 的二次型，状态约束 $C\mathbf y_k\in\mathcal X$ 仍是线性不等式。对比之下，$\hl{Alin}{A_{\rm lin}}=\partial F/\partial\mathbf x|_{\mathbf 0}$ 只在原点附近准；“不提升”的 EDMD 是对整个数据范围的最佳线性拟合，在原点附近反而不如 $A_{\rm lin}$（FIG 3 第一个任务）。

Here the first two components of $\boldsymbol\psi$ are $x,\dot x$ themselves, so $C=[I\ 0]$. The closed form is the normal equations $[A\ B]^\top=(Z^\top Z)^{-1}Z^\top Y^+$, where each row of $Z$ is $[\boldsymbol\psi(\mathbf x_i)^\top,\mathbf u_i^\top]$. With a linear model, the condensing of Lesson 13.4 carries over unchanged: $\hat x_{k}=\mathbf c^\top A^k\mathbf y_0+\sum_{j\lt k}\mathbf c^\top A^{k-1-j}B\,u_j$, the cost is a quadratic form in $U$, and state constraints $C\mathbf y_k\in\mathcal X$ remain linear inequalities. By contrast $\hl{Alin}{A_{\rm lin}}=\partial F/\partial\mathbf x|_{\mathbf 0}$ is accurate only near the origin, and EDMD "without lifting" is the best linear fit over the whole data range, which is actually worse than $A_{\rm lin}$ near the origin (FIG 3's first task).

高斯过程回归Gaussian-process regression

21.3 节的贝叶斯线性回归用有限个高斯鼓包当特征；鼓包无限多、铺满整个空间时，就得到高斯过程。给定数据 $X=\{\mathbf z_i\}$、带噪声 $\sigma_n$ 的观测 $\mathbf y$、核 $k(\mathbf z,\mathbf z')=\sigma_f^2\exp(-\tfrac12\sum_j(z_j-z_j')^2/\ell_j^2)$，在新点 $\mathbf z$ 的预测是

The Bayesian linear regression of Lesson 21.3 uses finitely many Gaussian bumps as features; with infinitely many bumps covering the whole space you get a Gaussian process. Given data $X=\{\mathbf z_i\}$, observations $\mathbf y$ with noise $\sigma_n$, and the kernel $k(\mathbf z,\mathbf z')=\sigma_f^2\exp(-\tfrac12\sum_j(z_j-z_j')^2/\ell_j^2)$, the prediction at a new point $\mathbf z$ is

$$ \hl{gpmean}{\mu(\mathbf z)}=m_0+\mathbf k_*^\top(K+\sigma_n^2I)^{-1}(\mathbf y-m_0),\qquad \sigma^2(\mathbf z)=k(\mathbf z,\mathbf z)-\mathbf k_*^\top(K+\sigma_n^2I)^{-1}\mathbf k_*,\qquad (\mathbf k_*)_i=k(\mathbf z,\mathbf z_i) $$

离所有数据都远时 $\mathbf k_*\to\mathbf 0$，于是 $\mu\to m_0$（先验均值），$\sigma\to\sigma_f$（先验标准差）。这两条性质就是本节的全部用处：FIG 4、FIG 5 的 $\sigma$ 标出没数据的地方；FIG 6 取 $m_0=0$，所以残差模型在没数据的地方退回到物理模型。代价是计算：求逆是 $O(n^3)$，每次预测 $O(n)$（均值）或 $O(n^2)$（方差），所以实时 GP-MPC 都用稀疏近似（几十到几百个诱导点）。本页 FIG 4 的超参数 $\sigma_f=0.4,\ \ell=0.9$ m，FIG 5 为 $\sigma_f=0.3$ m/s²、$\ell=0.35$ m，FIG 6 为 $\sigma_f=3$ m/s²、$\ell=2.5$ m/s，且只保留最多 150 个数据点；都是手选的，没有用边际似然优化。

Far from all data, $\mathbf k_*\to\mathbf 0$, so $\mu\to m_0$ (the prior mean) and $\sigma\to\sigma_f$ (the prior standard deviation). These two properties are all this lesson needs: in FIGs 4 and 5, $\sigma$ marks where there are no data; FIG 6 uses $m_0=0$, so the residual model falls back to the physics model where there are no data. The price is computation: the inverse costs $O(n^3)$ and each prediction $O(n)$ (mean) or $O(n^2)$ (variance), which is why real-time GP-MPC uses sparse approximations (tens to hundreds of inducing points). On this page the hyperparameters are $\sigma_f=0.4,\ \ell=0.9$ m in FIG 4, $\sigma_f=0.3$ m/s² and $\ell=0.35$ m in FIG 5, and $\sigma_f=3$ m/s² and $\ell=2.5$ m/s in FIG 6, which keeps at most 150 data points; all were chosen by hand rather than by maximizing the marginal likelihood.

沿时域传播不确定性，收紧约束Propagating uncertainty along the horizon and tightening constraints

FIG 5 的模型是 $\mathbf x_{k+1}=A\mathbf x_k+B u_k+B_d\,d(p_k)$，$\mathbf x=(p,v)$，$B_d=[\tfrac12\Delta t^2,\ \Delta t]^\top$。真实轨迹与均值的偏差 $\mathbf e_k$ 在一个预稳定反馈 $u=\bar u+K\mathbf e$（$K=[-1.2,\,-1.8]$）下近似满足（这个反馈只用来传播方差；闭环里真正的反馈是每周期重新规划）

FIG 5's model is $\mathbf x_{k+1}=A\mathbf x_k+B u_k+B_d\,d(p_k)$ with $\mathbf x=(p,v)$ and $B_d=[\tfrac12\Delta t^2,\ \Delta t]^\top$. The deviation $\mathbf e_k$ of the true trajectory from the mean, under a pre-stabilizing feedback $u=\bar u+K\mathbf e$ ($K=[-1.2,\,-1.8]$), approximately obeys (this feedback is used only to propagate the variance; the actual feedback in the closed loop is re-planning every cycle)

$$ \mathbf e_{k+1}\approx\underbrace{\big(A+BK+B_d\,[\mu_d'(\bar p_k)\ \ 0]\big)}_{\tilde A_k}\mathbf e_k+B_d\,\delta_k,\qquad \delta_k\sim\mathcal N\!\big(0,\sigma_d^2(\bar p_k)\big) $$

Hewing 等人假设各步的 $\delta_k$ 相互独立，于是 $\Sigma_{k+1}=\tilde A_k\Sigma_k\tilde A_k^\top+B_d\sigma_d^2B_d^\top$。可 GP 的误差是位置的函数，相邻几步的 $\bar p_k$ 很接近，误差几乎是同一个值；当成独立会低估累积。本页取另一个极端：把它们当作完全相关（同一个未知偏差），标准差直接线性累加 $\mathbf s_{k+1}=\tilde A_k\mathbf s_k+B_d\,\sigma_d(\bar p_k)$，$\sigma_{p,k}=|s_{k,1}|$。这是一个偏保守的上界。机会约束 $\Pr(p_k\le0)\ge1-\varepsilon$ 在高斯近似下变成

Hewing et al. assume the $\delta_k$ at different steps are independent, giving $\Sigma_{k+1}=\tilde A_k\Sigma_k\tilde A_k^\top+B_d\sigma_d^2B_d^\top$. But the GP's error is a function of position, and over a few steps the $\bar p_k$ are close, so the error is almost the same value; treating it as independent underestimates the accumulation. This page takes the other extreme and treats them as fully correlated (one unknown offset), so the standard deviations add up linearly: $\mathbf s_{k+1}=\tilde A_k\mathbf s_k+B_d\,\sigma_d(\bar p_k)$, $\sigma_{p,k}=|s_{k,1}|$. That is a somewhat conservative upper bound. Under a Gaussian approximation the chance constraint $\Pr(p_k\le0)\ge1-\varepsilon$ becomes

$$ \bar p_k+\hl{tube}{\kappa\,\sigma_{p,k}}\le0,\qquad \kappa=\Phi^{-1}(1-\varepsilon)\quad(\kappa=2\Rightarrow\varepsilon\approx2.3\%,\ \ \kappa=3\Rightarrow\varepsilon\approx0.13\%) $$

它在 FIG 5 里作为软约束（罚 $400\,v^2+20\,v$，$v$ 是越界量）进入 CEM 的代价。注意这里的 $\varepsilon$ 是每一步的违反概率，而且只有在 GP 的不确定性本身校准良好（21.3 节）时才是真的概率。

In FIG 5 it enters CEM's cost as a soft constraint (penalty $400\,v^2+20\,v$, where $v$ is the violation). Note that $\varepsilon$ is the violation probability per step, and it is a true probability only if the GP's uncertainty is itself well calibrated (Lesson 21.3).

残差模型Residual models

$$ \mathbf x_{k+1}=f_{\rm phys}(\mathbf x_k,\mathbf u_k)+B_d\,\hl{res}{\mathbf r_\phi}(\mathbf z_k),\qquad \mathbf r_\phi\ \text{fitted to}\ \ \mathbf r_i=B_d^{+}\big(\mathbf x_i^+-f_{\rm phys}(\mathbf x_i,\mathbf u_i)\big) $$

$\mathbf z$ 是残差依赖的那几个量（FIG 6 里是速度），往往比完整状态低维得多，所以需要的数据少。把它放进二次规划的常用办法是：在上一周期计划（平移后）的预测轨迹上计算 $\mathbf r_\phi$，把它当作已知的时变加速度加进线性预测，每周期重新计算。这和 13.4 节的实时迭代是一个思路：模型的非线性部分在上一个解附近“冻结”，每个周期只解一个 QP。

$\mathbf z$ are the quantities the residual depends on (velocity, in FIG 6), often far fewer than the full state, so little data is needed. A common way to put it into a quadratic program: evaluate $\mathbf r_\phi$ along the predicted trajectory of the (shifted) previous plan, add it to the linear prediction as a known time-varying acceleration, and recompute it every cycle. This is the same idea as the real-time iteration of Lesson 13.4: the model's nonlinear part is "frozen" around the previous solution, and each cycle solves just one QP.

7机器人中的它：用学到的残差高速飞行In robotics: flying fast with a learned residual

Torrente 等人（2021）的做法很有代表性：四旋翼的标称模型是刚体动力学，高速时的空气动力效应用 GP 从飞行数据里学成残差加速度，再接进非线性 MPC。下图是它的平面简化版：一个质点“无人机” $\ddot{\mathbf p}=\mathbf u+\mathbf d(\mathbf v)$，$|u_x|,|u_y|\le25$ m/s²，真实的阻力 $\mathbf d(\mathbf v)=-0.12\,\|\mathbf v\|\mathbf v-0.15\,\mathbf v$，标称模型不知道它。它要以给定速度沿半径 5 m 的圆飞一圈。控制器是线性 MPC（时域 20 步 = 1 s，x、y 两个方向各一个 20 变量的盒约束 QP）；“物理 + GP 残差”模式按第 6 节的办法把 GP 预测的加速度沿预测轨迹加进去。每飞完一圈，沿途测到的残差（$\Delta\mathbf v/\Delta t-\mathbf u$，带 0.1 m/s² 的噪声）加入数据。24.1 节的 FIG 5 已经在开环预测里比较过残差网络和黑箱网络：残差更省数据，出了数据范围有物理兜底。这里不再重复那个比较，而是看残差放进 MPC 闭环之后的事：它怎样作为已知的时变加速度进入每周期的 QP，跟踪误差降多少，以及 GP 的先验均值 0 怎样决定它在没飞过的速度上的行为。

The approach of Torrente et al. (2021) is representative: the quadrotor's nominal model is rigid-body dynamics, the aerodynamic effects at high speed are learned from flight data by a GP as a residual acceleration, and the result is plugged into nonlinear MPC. The figure is a planar simplification: a point-mass "drone" $\ddot{\mathbf p}=\mathbf u+\mathbf d(\mathbf v)$ with $|u_x|,|u_y|\le25$ m/s² and a true drag $\mathbf d(\mathbf v)=-0.12\,\|\mathbf v\|\mathbf v-0.15\,\mathbf v$ that the nominal model does not know. It must fly one lap of a circle of radius 5 m at a given speed. The controller is a linear MPC (horizon 20 steps = 1 s, one 20-variable box-constrained QP for each of x and y); the "physics + GP residual" mode adds the GP's predicted acceleration along the predicted trajectory as in Section 6. After each lap, the residuals measured along the way ($\Delta\mathbf v/\Delta t-\mathbf u$, with 0.1 m/s² of noise) are added to the data. FIG 5 of Lesson 24.1 already compared a residual network with a black-box network in open-loop prediction: the residual needs less data and has physics to fall back on outside the data. Rather than repeat that comparison, this figure looks at what happens once the residual is inside the MPC loop: how it enters each cycle's QP as a known time-varying acceleration, how much tracking error it removes, and how the GP's prior mean of 0 decides its behaviour at speeds it has never flown.

FIG 6残差学习：只学物理模型漏掉的阻力Residual learning: learn only the drag the physics model misses

俯视 (x, y)TOP VIEW (x, y)

速度平面：数据与 GP 残差VELOCITY PLANE · data and GP residual

左：琥珀色虚线是参考圆，蓝色是这一圈的飞行轨迹（误差放大 5 倍画出，便于看清），点是无人机此刻的位置。右：速度平面 $(v_x,v_y)$，绿色小点是数据，底色越青 = GP 越没把握，紫色箭头是 GP 在网格上预测的残差加速度（缩小画出），红色箭头是同一处的真实阻力，琥珀色虚线圆是当前设定的速度。RMSE 统计第 1 s 之后的位置误差。“物理 + 真实阻力”是知道真实阻力时 MPC 能做到的下限。 Left: the dashed amber circle is the reference and blue is this lap's flight path (its error drawn 5× enlarged so that it is visible); the dot is the drone's current position. Right: the velocity plane $(v_x,v_y)$; the small green dots are the data, more cyan shading = the GP is less sure, the purple arrows are the GP's predicted residual acceleration on a grid (drawn scaled down), the red arrows the true drag at the same points, and the dashed amber circle the speed currently set. The RMSE counts position error after the first second. "Physics + true drag" is the floor MPC can reach when it knows the true drag.

试一试Try this

按“重置数据”，在 8 m/s 下用“物理”飞一圈（RMSE 0.153 m，阻力把它往圆内拖），再换“物理 + GP 残差”以同样速度飞一圈：误差降到物理模型的 30% 以下（约 0.02 m）。Press "Reset data", fly one lap at 8 m/s with "physics" (RMSE 0.153 m; drag pulls the drone inside the circle), then switch to "physics + GP residual" and fly a lap at the same speed: the error drops below 30% of the physics model's (to about 0.02 m).
接着把速度调到 10 m/s 以上，用“物理 + GP 残差”飞：只要数据都来自比当前慢 3 m/s 以上的飞行，残差帮的忙就很小（误差仍超过物理模型的 60%；先飞 8 m/s 再飞 11 m/s 时 0.297 对 0.363 m）。在速度平面里看：这个速度圈落在青色区域，GP 退回了先验 0，也就是退回了物理模型。再飞一圈，误差就降下来（0.115 m）。Next set the speed above 10 m/s and fly with "physics + GP residual": as long as all data come from flights at least 3 m/s slower, the residual helps little (the error stays above 60% of the physics model's; 0.297 versus 0.363 m at 11 m/s after laps at 8 m/s). Look at the velocity plane: this speed circle lies in the cyan region, where the GP has fallen back to its prior of 0, i.e. back to the physics model. Fly another lap and the error comes down (0.115 m).

真实系统里In real systems

神经网络模型 + 采样 MPC：Williams 等人（ICRA 2017）把多层网络学到的车辆动力学放进 MPPI，在 1/5 比例的 AutoRally 越野车上做激进驾驶；Nagabandi 等人（ICRA 2018）用网络动力学加随机打靶的 MPC 控制模拟机器人和一台真实的小型足式机器人，并用它初始化无模型 RL（这类方法在 24.2 节展开）。GP 残差：Hewing、Kabzan 与 Zeilinger 的谨慎 MPC（IEEE TCST 2020）在 1:43 微型赛车上以 20 ms 采样周期运行；Kabzan 等人（RA-L 2019）把同一思路用在 AMZ Driverless 的全尺寸方程式赛车上，用稀疏 GP 在线学车辆模型的残差，随着数据积累，单圈用时缩短。Torrente 等人（RA-L 2021）用 GP 学四旋翼的空气动力残差，在最高 14 m/s 的飞行里把跟踪误差降低了最多 70%。神经网络残差：Salzmann 等人的 Real-time Neural MPC（RA-L 2023）把大得多的网络残差放进四旋翼的 MPC，在嵌入式平台上以 50 Hz 运行，位置跟踪误差比不带网络的 MPC 最多降低 82%。Koopman：Korda 与 Mezić（Automatica 2018）提出了带输入的 EDMD 与线性 MPC；Bruder 等人（IEEE T-RO 2021）用 Koopman 模型和线性 MPC 控制气动软体机械臂。价值 + 学到的模型：TD-MPC 在学到的潜模型里用 MPPI 规划，并以学到的价值函数作终端代价（24.3 节），这正是 13.4 节“终端代价代替时域之外的未来”在学习框架里的对应物。

Neural-network models with sampling MPC: Williams et al. (ICRA 2017) put vehicle dynamics learned by a multilayer network into MPPI for aggressive driving on a 1/5-scale AutoRally off-road car; Nagabandi et al. (ICRA 2018) used network dynamics with random-shooting MPC to control simulated robots and a real small legged robot, and used it to initialize model-free RL (methods of this kind are developed in Lesson 24.2). GP residuals: the cautious MPC of Hewing, Kabzan and Zeilinger (IEEE TCST 2020) runs on 1:43-scale miniature race cars with a 20 ms sampling time; Kabzan et al. (RA-L 2019) applied the same idea to AMZ Driverless's full-size Formula Student car, learning the residual of the vehicle model online with a sparse GP, and lap times fell as data accumulated. Torrente et al. (RA-L 2021) learned the quadrotor's aerodynamic residual with a GP and reduced tracking error by up to 70% in flights at up to 14 m/s. Neural-network residuals: the Real-time Neural MPC of Salzmann et al. (RA-L 2023) puts much larger network residuals into quadrotor MPC, runs at 50 Hz on an embedded platform, and reduces position tracking error by up to 82% compared with MPC without the network. Koopman: Korda and Mezić (Automatica 2018) introduced EDMD with inputs together with linear MPC; Bruder et al. (IEEE T-RO 2021) controlled a pneumatic soft robot arm with a Koopman model and linear MPC. Value + learned model: TD-MPC plans with MPPI in a learned latent model and uses a learned value function as the terminal cost (Lesson 24.3), the learning-based counterpart of Lesson 13.4's "the terminal cost stands in for the future beyond the horizon".

什么时候“经典 MPC + 学到的残差”胜过端到端强化学习When "classical MPC + a learned residual" beats end-to-end RL

端到端强化学习（23.1 节）直接学一个从观测到动作的策略，部署时只要一次网络前向。MPC + 残差则保留了一个可解释的优化问题。两者各有所长，下表是一个经验性的对照：

End-to-end reinforcement learning (Lesson 23.1) learns a policy from observations to actions directly, and at deployment it needs one network forward pass. MPC with a residual keeps an interpretable optimization problem. Each has its strengths; the table is a rule-of-thumb comparison:

情形Situation	倾向Leans towards	原因Why
物理模型基本对，只差一部分（阻力、轮胎、摩擦）The physics model is mostly right and misses one part (drag, tyres, friction)	MPC + 残差MPC + residual	残差低维、数据需求小（FIG 6 一圈数据就够），没数据处还能退回物理模型The residual is low-dimensional and needs little data (one lap was enough in FIG 6), and falls back to physics where there are no data
硬约束必须满足、任务和约束经常变Hard constraints must hold, and tasks and constraints change often	MPC + 残差MPC + residual	约束和代价写在优化问题里，换一个参考轨迹或障碍物不需要重新训练Constraints and cost live in the optimization problem; a new reference or obstacle needs no retraining
接触丰富、模型很难写（灵巧手、复杂地形上的行走）Contact-rich and hard to model (dexterous hands, walking over rough terrain)	端到端 RL（仿真里训练）End-to-end RL (trained in simulation)	在线优化穿过接触切换很难；RL 把这份难度挪到离线的大规模仿真里Optimizing online through contact switches is hard; RL moves that difficulty into large-scale offline simulation
机载算力很小、控制频率很高Very little onboard compute, very high control rate	端到端 RLEnd-to-end RL	一次网络前向通常比每周期解一个优化问题便宜得多One network forward pass is usually far cheaper than solving an optimization problem every cycle
真正的目标难以写成跟踪误差（例如“最快通过所有门”）The true objective is hard to express as a tracking error ("pass all gates as fast as possible")	端到端 RLEnd-to-end RL	RL 可以直接优化任务层面的目标RL can optimize the task-level objective directly

最后一行有一个很好的例子。Song 等人（Science Robotics 2023）在无人机竞速上系统比较了最优控制与强化学习，结论是 RL 的优势不在于把目标优化得更好，而在于它优化的是一个更好的目标：最优控制把任务拆成“先规划轨迹、再跟踪轨迹”，跟踪误差这一目标限制了控制器能表现的行为，遇到没建模的效应时表现更差；RL 直接优化任务层面的目标。同一个组的 Swift（Kaufmann 等，Nature 2023）就是用深度强化学习训练的竞速无人机，在比赛中赢过了人类冠军。所以“MPC + 残差”的优势主要在数据效率、约束和可改性，而不是在所有任务上都更强。两条路也在合流：用 RL 学的价值函数做 MPC 的终端代价（TD-MPC），或者用 MPC 生成的数据去训练策略。

The last row has a good example. Song et al. (Science Robotics 2023) compared optimal control and reinforcement learning systematically on drone racing and concluded that RL's advantage is not that it optimizes its objective better, but that it optimizes a better objective: optimal control splits the task into "plan a trajectory, then track it", the tracking-error objective limits the behaviours the controller can express, and it performs worse when facing unmodelled effects; RL optimizes the task-level objective directly. Swift (Kaufmann et al., Nature 2023), from the same group, is a racing drone trained with deep RL that beat human champions in races. So the advantage of "MPC + residual" lies mainly in data efficiency, constraints and adaptability, not in being stronger at every task. The two roads are also merging: a value function learned with RL as MPC's terminal cost (TD-MPC), or data generated by MPC used to train a policy.

8常见误区Common pitfalls

误区 1：训练损失低，MPC 就会好Pitfall 1: a low training loss means a good MPC

FIG 1 里只见过下方数据的网络，训练集损失（$1.5\times10^{-4}$）比全数据网络（$1.8\times10^{-3}$）低一个数量级，在它的数据区域里也更准，可它让 MPC 一次也没竖起来。评估学到的模型要在闭环会访问的状态上测，最好是多步预测误差（24.1 节），而不是在训练分布上的一步误差。数据还要随着控制器变好而更新：控制器一变，访问的状态就变了（DAgger 式的迭代采集）。

In FIG 1 the bottom-only network ends with a training-set loss ($1.5\times10^{-4}$) an order of magnitude below the whole-plane network's ($1.8\times10^{-3}$) and is more accurate inside its data region, yet with it the MPC never gets upright. Evaluate a learned model on the states the closed loop will visit, preferably by multi-step prediction error (Lesson 24.1), not by one-step error on the training distribution. And refresh the data as the controller improves: a new controller visits new states (DAgger-style iterative collection).

误区 2：模型可导，就一定用梯度Pitfall 2: the model is differentiable, so always use gradients

FIG 2 里梯度在光滑代价上用三分之一的计算就赢了 MPPI，但它看不见禁区。网络的梯度还有两个隐患：在没数据的地方，网络对输入的导数可以任意大或任意错（它只被训练成拟合函数值，不是拟合导数）；ReLU 网络的导数是分段常数。梯度把规划器沿着这些错误的斜率推向没数据的地方，正是“利用模型误差”的最快通道。

In FIG 2 gradients beat MPPI with a third of the compute on a smooth cost, but they cannot see the no-go zone. Network gradients have two more hidden dangers: where there are no data, a network's derivative with respect to its input can be arbitrarily large or wrong (it was trained to fit function values, not derivatives); and a ReLU network's derivative is piecewise constant. Gradients push the planner along these wrong slopes towards data-free regions, the fastest route to "exploiting model error".

误区 3：Koopman 模型是精确的线性化Pitfall 3: a Koopman model is an exact linearization

只有在提升函数张成一个对动力学不变的子空间时，EDMD 才是精确的；实际中几乎从不成立，所以预测误差随时域累积（FIG 3：40 步时所有提升的误差都接近 0.44）。提升维数越高，最小二乘越容易过拟合，状态约束 $C\mathbf y\in\mathcal X$ 也只是在预测的 $\hat{\mathbf x}$ 上成立。要把 Koopman MPC 的时域选在预测还可信的范围内，并像普通 MPC 一样为模型误差留余量。

EDMD is exact only when the lifting functions span a subspace that the dynamics leave invariant, which almost never happens in practice, so prediction error accumulates with the horizon (FIG 3: at 40 steps every lift's error is close to 0.44). The higher the lifted dimension, the easier the least squares overfits, and state constraints $C\mathbf y\in\mathcal X$ hold only for the predicted $\hat{\mathbf x}$. Choose the Koopman MPC's horizon within the range where predictions are still trustworthy, and leave a margin for model error as with any MPC.

误区 4：不确定性一定要“越保守越好”Pitfall 4: with uncertainty, "the more conservative the better"

FIG 5 里 $\kappa=3$ 第一趟比 $\kappa=2$ 离墙更远，也更早到目标（6.1 s 对 7.1 s），但这不是规律：太大的 $\kappa$ 或 $\beta$ 会让控制器永远不进入没数据的区域，于是那里永远没有数据，不确定性永远不会下降（FIG 4 的“绕行”路线就再也学不到中间的地有多好）。安全的学习需要有意识地在可控的风险下探索：在确定安全的集合里逐步扩张（安全贝叶斯优化、安全集方法），而不是只把余量调大。

In FIG 5, $\kappa=3$ stays further from the wall on the first run than $\kappa=2$ and even reaches the target sooner (6.1 s versus 7.1 s), but that is no rule: a $\kappa$ or $\beta$ that is too large keeps the controller out of data-free regions forever, so those regions never get data and their uncertainty never falls (FIG 4's "detour" never learns how good the ground in the middle is). Safe learning needs deliberate exploration at a controlled risk, expanding a set known to be safe step by step (safe Bayesian optimization, safe-set methods), not just a bigger margin.

误区 5：残差可以用任何模型来学Pitfall 5: any model will do for the residual

残差模型在没数据的地方应该“闭嘴”，也就是输出接近 0，让物理模型做主。GP（先验均值 0）天然如此（FIG 6 第二个任务；24.1 节 FIG 5 里的物理残差网络在数据外也靠物理模型兜底，但网络本身的输出并不会自动归零）。普通的 tanh 或 ReLU 网络在训练范围外会按自己的方式外推，可能把一个本来还算准的物理模型带偏；用网络时要配合不确定性估计（集成，21.3 节）、权重衰减，或把输出乘一个随“离数据距离”衰减的门控。另外，残差只能修正物理模型的状态和输入能解释的误差：如果误差来自没测到的量（风、载荷），它只会学到一个平均值。

A residual model should "stay quiet" where there are no data, i.e. output nearly 0 and let the physics model decide. A GP with prior mean 0 does this by construction (FIG 6's second task; the residual network of Lesson 24.1's FIG 5 also leans on physics outside its data, but the network's own output does not go to zero by itself). An ordinary tanh or ReLU network extrapolates in its own way outside its training range and can pull a physics model that was reasonably right off course; with networks, pair the residual with an uncertainty estimate (ensembles, Lesson 21.3), weight decay, or a gate on the output that decays with the distance to the data. Also, a residual can only correct errors that the physics model's states and inputs can explain: if the error comes from something unmeasured (wind, payload), it will only learn an average.

9小练习Check yourself

1. 一个学到的动力学模型在验证集上的一步预测误差很小，可接进 MPC 后闭环表现很差，规划的轨迹在模型里看起来很好。最可能的原因是什么？1. A learned dynamics model has a small one-step prediction error on the validation set, yet in closed loop with MPC it performs badly, while the planned trajectories look great inside the model. What is the most likely cause?

选项 B。验证集与训练集同分布，测的是模型在“数据去过的地方”的误差；而优化器专门寻找预测代价低的计划，模型误差偏乐观的地方最容易被选中（第 6 节的“优化者的诅咒”）。“模型里看起来很好”正是这个特征。FIG 1 的窄数据网络在自己的数据区域比全数据网络还准，却让 MPC 失败。欠拟合（C）会让验证误差也大；只执行第一步（D）是 MPC 的反馈来源，不是问题所在。Option B. The validation set has the training distribution, so it measures error where the data have been; the optimizer specifically looks for plans with a low predicted cost, so places where the model errs optimistically are the most likely to be chosen (the "optimizer's curse" of Section 6). "Looks great inside the model" is exactly the symptom. FIG 1's narrow-data network is more accurate in its own data region than the whole-plane one and still makes MPC fail. Underfitting (C) would make the validation error large too; applying only the first step (D) is where MPC's feedback comes from, not the problem.

2. 在一个学到的神经网络动力学模型上做 MPC，代价里有一项“碰到障碍物罚 100”（0/1 指示函数）。下面哪种规划器最合适？2. You run MPC on a learned neural-network dynamics model, and the cost includes "100 for touching an obstacle" (a 0/1 indicator). Which planner fits best?

选项 C。网络可导，但代价不可导：指示函数的梯度几乎处处为 0，梯度法感觉不到障碍（FIG 2 打开禁区后，梯度法在预算 900 时有 13 个周期落在禁区里，MPPI 一次也没有）。采样法只需要代价值；或者把代价改成光滑的（例如到障碍的距离的罚函数）再用梯度。指示函数不是凸的（D 错），原点线性化对远离原点的轨迹不准（B 错）。Option C. The network is differentiable but the cost is not: the indicator's gradient is zero almost everywhere, so gradient descent cannot feel the obstacle (with the zone on in FIG 2 at budget 900, the gradient planner spent 13 cycles inside it and MPPI none). Sampling needs only cost values; alternatively make the cost smooth (a penalty on the distance to the obstacle, say) and then use gradients. The indicator is not convex (D is wrong), and a linearization at the origin is inaccurate for trajectories far from it (B is wrong).

3. 用 EDMD 学到了提升模型 $\mathbf y^+=A\mathbf y+B\mathbf u$，$\mathbf y=\boldsymbol\psi(\mathbf x)$，提升维数 32。在它上面做 MPC，下列哪个说法正确？3. EDMD has produced a lifted model $\mathbf y^+=A\mathbf y+B\mathbf u$ with $\mathbf y=\boldsymbol\psi(\mathbf x)$ and lifted dimension 32. Which statement about MPC on it is correct?

选项 A。在提升空间里模型是线性的，所以 13.4 节的缩并照搬，QP 是凸的；缩并后决策变量只有 $N\times m$ 个，32 维的提升状态只影响构造矩阵的开销。可提升只是近似（FIG 3 中 40 步预测误差约 0.44），所以 C 错，仍需每个周期从测量重新提升（这是反馈）。预测时不需要在每一步重新提升，正是“线性推进”让问题保持为 QP（D 错）。Option A. The model is linear in the lifted space, so the condensing of Lesson 13.4 carries over and the QP is convex; after condensing there are only $N\times m$ decision variables, and the 32-dimensional lifted state only affects the cost of building the matrices. But lifting is an approximation (40-step prediction error about 0.44 in FIG 3), so C is wrong, and each cycle still re-lifts from the measurement (that is the feedback). Prediction does not re-lift at every step: advancing linearly is precisely what keeps the problem a QP (D is wrong).

4. GP-MPC 把状态约束 $p_k\le0$ 收紧成 $\bar p_k+\kappa\sigma_{p,k}\le0$。数据越来越多时，会发生什么？4. GP-MPC tightens the state constraint $p_k\le0$ into $\bar p_k+\kappa\sigma_{p,k}\le0$. What happens as more and more data arrive?

选项 D。$\kappa$ 是设计者选的置信水平，不会变（A、B 错）。$\sigma_{p,k}$ 由 GP 的预测标准差沿时域传播而来，数据越多，有数据处的认知不确定性越小（21.3 节），只剩观测噪声决定的下限。FIG 5 第二趟（$\kappa=2$）就是这样：管道变窄，到达目标的时间从 7.1 s 缩短到约 4.5 s。Option D. $\kappa$ is the confidence level the designer chose and does not change (A and B are wrong). $\sigma_{p,k}$ is the GP's predictive standard deviation propagated along the horizon, and with more data the epistemic uncertainty falls where the data are (Lesson 21.3), down to a floor set by the observation noise. That is FIG 5's second run ($\kappa=2$): the tube narrows and the time to reach the target falls from 7.1 s to about 4.5 s.

5. 一台四旋翼的刚体模型已经相当准，只有高速时的空气阻力没建模；要在一周的实验时间里显著提高高速跟踪精度，并且必须满足推力上限和禁飞区。哪种方案最合适？5. A quadrotor's rigid-body model is already quite accurate; only the air drag at high speed is unmodelled. You have one week of experiments to substantially improve high-speed tracking, and thrust limits and no-fly zones must be respected. Which approach fits best?

选项 B。物理模型已经解释了大部分动力学，残差只依赖速度等少数几个量，所需数据少（FIG 6 里一圈数据就把 8 m/s 的误差降到约 0.02 m），没数据处又会退回物理模型；推力上限和禁飞区直接写成 MPC 的约束。Torrente 等人（2021）和 Salzmann 等人（2023）就是这条路线。从零学完整动力学（C）要多得多的数据，外推也没有物理兜底；在真机上从零训练 RL（A）一周内很难做到，也难以保证约束；加长时域（D）不能修正模型误差，反而让预测更远、误差更大。Option B. The physics model already explains most of the dynamics, the residual depends on only a few quantities such as velocity and needs little data (one lap brought the 8 m/s error down to about 0.02 m in FIG 6), and it falls back to physics where there are no data; thrust limits and no-fly zones go straight into MPC as constraints. This is the route of Torrente et al. (2021) and Salzmann et al. (2023). Learning the full dynamics from scratch (C) needs far more data and has no physics to fall back on when extrapolating; training RL from scratch on the real drone (A) is hard to do in a week and hard to make respect constraints; a longer horizon (D) does not correct model error and only predicts further ahead with larger errors.

10延伸阅读Further reading

L. Hewing, K. P. Wabersich, M. Menner & M. N. Zeilinger, Learning-Based Model Predictive Control: Toward Safe Learning in Control, Annual Review of Control, Robotics, and Autonomous Systems 3, 2020 — 学习型 MPC 的综述：学模型、学代价、安全性A survey of learning-based MPC: learning the model, learning the cost, and safety
L. Hewing, J. Kabzan & M. N. Zeilinger, Cautious Model Predictive Control Using Gaussian Process Regression, IEEE Transactions on Control Systems Technology 28(6), 2020 — GP 残差、方差沿时域的传播与机会约束（FIG 5 的原型）GP residuals, variance propagation along the horizon and chance constraints (the prototype of FIG 5)
J. Kabzan, L. Hewing, A. Liniger & M. N. Zeilinger, Learning-Based Model Predictive Control for Autonomous Racing, IEEE Robotics and Automation Letters 4(4), 2019 — 全尺寸赛车上在线学习 GP 残差Online GP residual learning on a full-size race car
G. Torrente, E. Kaufmann, P. Föhn & D. Scaramuzza, Data-Driven MPC for Quadrotors, IEEE Robotics and Automation Letters 6(2), 2021 — 四旋翼空气动力残差的 GP 学习（FIG 6 的原型）GP learning of quadrotor aerodynamic residuals (the prototype of FIG 6)
T. Salzmann, E. Kaufmann, J. Arrizabalaga, M. Pavone, D. Scaramuzza & M. Ryll, Real-Time Neural MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms, IEEE Robotics and Automation Letters 8(4), 2023 — 把大型神经网络放进实时 MPCLarge neural networks inside real-time MPC
M. Korda & I. Mezić, Linear Predictors for Nonlinear Dynamical Systems: Koopman Operator Meets Model Predictive Control, Automatica 93, 2018 — 带输入的 EDMD 与 Koopman MPCEDMD with inputs and Koopman MPC
D. Bruder, X. Fu, R. B. Gillespie, C. D. Remy & R. Vasudevan, Data-Driven Control of Soft Robots Using Koopman Operator Theory, IEEE Transactions on Robotics, 2021 — 软体机械臂上的 Koopman MPCKoopman MPC on a soft robot arm
G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots & E. A. Theodorou, Information Theoretic MPC for Model-Based Reinforcement Learning, ICRA 2017 — 神经网络动力学 + MPPI，激进驾驶Neural-network dynamics with MPPI for aggressive driving
A. Nagabandi, G. Kahn, R. S. Fearing & S. Levine, Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning, ICRA 2018 — 网络动力学 + 随机打靶 MPCNetwork dynamics with random-shooting MPC
Y. Song, A. Romero, M. Müller, V. Koltun & D. Scaramuzza, Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning, Science Robotics 8(82), 2023 — 最优控制与强化学习在无人机竞速上的系统比较A systematic comparison of optimal control and RL in drone racing
C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for Machine Learning, MIT Press, 2006 — 高斯过程回归的标准参考The standard reference for Gaussian-process regression
J. B. Rawlings, D. Q. Mayne & M. M. Diehl, Model Predictive Control: Theory, Computation, and Design, 2nd ed., Nob Hill Publishing, 2017 — 鲁棒与随机 MPC 的理论背景Theory background for robust and stochastic MPC