学习动力学模型Learning Dynamics Models

Part IV 的控制器几乎都要一个模型：LQR（13.3 节）要 \(A,B\)，MPC（13.4 节）要在每个控制周期里把模型往前滚几十步。13.2 节从牛顿定律写出了摆的状态方程；8.3 节把多关节机械臂写成 \(M(\mathbf q)\ddot{\mathbf q}+\mathbf b=\boldsymbol\tau\)。这些方程都假设我们知道所有参数，而且所有力都在方程里。

真实机器人不是这样。减速器里的摩擦随温度变，夹爪上多了一个没称过的工件，腿踩在泥地上，手里揉着一块面团，空气在桨叶下面打旋。这些东西要么很难写成方程，要么方程里的参数测不准。另一条路是从数据里学：让机器人动一动，记录下大量的 \((s_t,a_t,s_{t+1})\)，训练一个函数 \(s_{t+1}\approx f_\theta(s_t,a_t)\)。这就是学习动力学模型，也是所有 model-based 强化学习（本章的 24.2、24.3、24.4 节）的第一步。

它看上去只是一个回归问题（16.1 节），可动力学模型的用法和普通回归不同：它的输出会被再喂回给它自己，一步接一步地滚动。一步上的一点点误差会被下一步当成输入，误差会累积，模型还会被带到训练数据没覆盖的状态。本节的大部分内容都是在处理这个“自己喂自己”带来的问题。

Nearly every controller of Part IV needs a model: LQR (Lesson 13.3) needs \(A,B\), and MPC (Lesson 13.4) rolls a model forward dozens of steps in every control period. Lesson 13.2 wrote the pendulum's state equation from Newton's law; Lesson 8.3 wrote a multi-joint arm as \(M(\mathbf q)\ddot{\mathbf q}+\mathbf b=\boldsymbol\tau\). Those equations assume we know every parameter and that every force is in the equation.

Real robots are not like that. Friction in a gearbox changes with temperature, the gripper carries a part nobody weighed, a leg steps into mud, a hand kneads dough, air swirls under a propeller. These are either hard to write as equations or their parameters cannot be measured well. The alternative is to learn from data: let the robot move, log many triples \((s_t,a_t,s_{t+1})\), and train a function \(s_{t+1}\approx f_\theta(s_t,a_t)\). This is learning a dynamics model, and it is the first step of all model-based reinforcement learning (Lessons 24.2, 24.3 and 24.4 of this chapter).

It looks like a plain regression problem (Lesson 16.1), but a dynamics model is used differently from an ordinary regressor: its output is fed back into itself, step after step. A small error at one step becomes the next step's input, errors pile up, and the model gets carried into states the training data never covered. Most of this lesson is about the problems this "feeding itself" creates.

本节的例子是一个单关节机械臂：电机通过减速器驱动一根带负载的杆，相当于 13.2 节里的摆加上一个力矩输入。状态 \(s=(\theta,\omega)\)：\(\theta\) 是从竖直向下量起的角度（rad），\(\omega\) 是角速度（rad/s）；动作 \(a=\tau\) 是电机力矩，限制在 ±2 N·m。“真实世界”（我们假装不知道它）是

The example throughout this lesson is a one-joint arm: a motor drives a loaded link through a gearbox, which is the pendulum of Lesson 13.2 with a torque input. The state is \(s=(\theta,\omega)\): \(\theta\) the angle measured from hanging straight down (rad) and \(\omega\) the angular velocity (rad/s); the action \(a=\tau\) is the motor torque, limited to ±2 N·m. The "real world" (which we pretend not to know) is

最后两项是黏性摩擦和（平滑过的）库仑摩擦。电机最多 2 N·m，小于把杆水平举住所需的 4.9 N·m，所以要把杆甩过头顶，只能来回“泵”能量。控制器每 \(\Delta t=0.05\) s（20 Hz）读一次状态、发一次力矩，力矩在一个周期内保持不变（13.2 节的零阶保持）。所以我们要学的是一个离散时间的映射：

The last two terms are viscous and (smoothed) Coulomb friction. The motor gives at most 2 N·m, less than the 4.9 N·m needed to hold the link horizontal, so swinging it over the top requires pumping energy back and forth. The controller reads the state and sends a torque every \(\Delta t=0.05\) s (20 Hz), and the torque is held over each period (the zero-order hold of Lesson 13.2). So what we learn is a discrete-time map:

训练数据是 40 段、每段 2.5 s 的随机运动：从 \(|\theta|<1.5\)、\(|\omega|<2\) 内的随机状态出发，每 0.2 s 换一个 [−2, 2] 内均匀随机的力矩，一共 2000 个转移。下图把模型画成相平面上的箭头：每个格点上，青色箭头是真实的一步变化 \((\Delta\theta,\Delta\omega)\)，琥珀色箭头是模型预测的。底色表示模型的一步误差。先按“训练”，再把探针拖到数据覆盖的区域内外比较。鼠标放在公式里的 \(f_\theta\) 上，琥珀色箭头会加粗。

The training data are 40 random motions of 2.5 s each: start from a random state with \(|\theta|<1.5\), \(|\omega|<2\) and switch every 0.2 s to a torque drawn uniformly from [−2, 2], for 2000 transitions in all. The figure below draws a model as arrows on the phase plane: at each grid point the cyan arrow is the true one-step change \((\Delta\theta,\Delta\omega)\) and the amber arrow the model's prediction. The shading shows the model's one-step error. Press "Train", then drag the probe inside and outside the region the data cover and compare. Hover \(f_\theta\) in the formula to thicken the amber arrows.

FIG 1 的网络预测的是变化量 \(\Delta s=s_{t+1}-s_t\)，而且输入、输出都做了归一化。这两个选择看似细节，却往往决定了模型能不能用。原因很简单：控制周期很短，下一刻的状态和这一刻几乎一样。直接预测 \(s_{t+1}\) 时，网络大部分容量都花在“把输入原样抄过去”上，而我们真正关心的那一小点变化，在损失里只占很小的比例。预测 \(\hl{delta}{\Delta s}\) 则把“抄过去”写死在结构里，网络只学变化本身：

FIG 1's network predicts the change \(\Delta s=s_{t+1}-s_t\), and both its inputs and outputs are normalized. These choices look like details but often decide whether the model is usable at all. The reason is simple: the control period is short, so the next state is almost the same as this one. When the network predicts \(s_{t+1}\) directly, most of its capacity goes into "copying the input through", and the small change we actually care about is a tiny part of the loss. Predicting \(\hl{delta}{\Delta s}\) builds the copying into the structure and leaves the network to learn only the change itself:

\(\mu_x,\sigma_x\) 是输入在数据上的均值和标准差，\(\mu_\Delta,\sigma_\Delta\) 是变化量的均值和标准差，\(\odot\) 表示逐分量相乘。归一化有两个作用：输入的各个分量尺度相近，优化更顺；输出的各个分量在损失里权重相当。不归一化时，\(\Delta\omega\) 的典型大小（0.58 rad/s）是 \(\Delta\theta\)（0.13 rad）的 4 倍多，平方误差里它占了约 20 倍的权重，角度那一维几乎没人管。下图同时训练四个只在这两点上不同的网络（结构、初值、数据、小批量顺序都相同），比较它们在测试数据上的相对一步误差。

\(\mu_x,\sigma_x\) are the inputs' mean and standard deviation over the data, \(\mu_\Delta,\sigma_\Delta\) those of the change, and \(\odot\) is elementwise multiplication. Normalization does two jobs: the input components get comparable scales, which makes optimization smoother, and the output components get comparable weight in the loss. Without it, the typical \(\Delta\omega\) (0.58 rad/s) is more than 4 times the typical \(\Delta\theta\) (0.13 rad), so it carries about 20 times the weight in the squared error and the angle is left almost unattended. The figure below trains four networks that differ only in these two choices (same architecture, initialization, data and minibatch order) and compares their relative one-step error on test data.

几点观察。(1) 归一化不是让模型“整体更好”，而是决定损失在乎哪一维：20 Hz 下，Δs 原始单位（蓝）的 ω 误差（2.4%）其实比归一化（绿，3.0%）还低一点，代价是 θ 误差几乎翻倍（3.1% 对 1.7%）。(2) 预测 s′ 时就算归一化（紫）也不行：归一化把 s′ 的整个范围（θ 的标准差 0.66 rad）缩成 1，可我们关心的变化只有它的五分之一。(3) 右图单条轨迹的滚动结果会随运气起伏，比较模型要看很多条轨迹，下一节专门讲这件事。

A few observations. (1) Normalization does not make the model "better overall"; it decides which dimension the loss cares about: at 20 Hz, Δs in raw units (blue) actually has a slightly lower ω error (2.4%) than the normalized version (green, 3.0%), at the price of nearly double the θ error (3.1% versus 1.7%). (2) Predicting s′ fails even when normalized (purple): normalization scales the whole range of s′ (θ has a standard deviation of 0.66 rad) to 1, while the change we care about is a fifth of that. (3) A rollout along a single trajectory, as on the right, depends on luck; comparing models needs many trajectories, which is what the next section is about.

规划器（13.4 节的 MPC、6.11 节的 CEM/MPPI）用模型的方式是开环滚动：给定起点和一串动作，\(\hat s_{t+1}=f_\theta(\hat s_t,a_t)\)，每一步都把自己上一步的预测当输入。一步误差 \(\varepsilon\) 很小，并不说明 H 步后误差也小：误差会被动力学放大，而一旦预测的状态离开数据，每一步的 \(\varepsilon\) 本身也变大。

怎么知道什么时候不能再信模型？21.3 节的办法：训练几个只有初值和小批量顺序不同的模型（集成），看它们的分歧 \(\hl{spread}{\sigma_{\mathrm{ens}}}\)。下图用 5 个一步模型从同一个起点、按同一串力矩滚动，与真实轨迹比较。

A planner (MPC of Lesson 13.4, CEM/MPPI of Lesson 6.11) uses a model by open-loop rollout: given a start and a sequence of actions, \(\hat s_{t+1}=f_\theta(\hat s_t,a_t)\), each step taking its own previous prediction as input. A small one-step error \(\varepsilon\) does not mean the error after H steps is small: errors get amplified by the dynamics, and once the predicted state leaves the data the per-step \(\varepsilon\) itself grows.

How do we know when to stop trusting the model? The approach of Lesson 21.3: train several models that differ only in initialization and minibatch order (an ensemble) and look at their disagreement \(\hl{spread}{\sigma_{\mathrm{ens}}}\). The figure below rolls 5 one-step models from the same start through the same torque sequence and compares them with the true trajectory.

“泵能量”的那条轨迹说明了 model-based 方法最大的麻烦：规划器正是会去找这种轨迹的。如果目标是把杆甩上去，优化器会在模型里寻找能量涨得最快的动作，而模型在高能量区没有数据，它的回答是乱猜，甚至可能比真实系统“更容易”甩上去。这叫模型被利用（model exploitation）。常见的对策都在本章后面：让轨迹样本在集成的各个成员中传播、对它们的代价取平均，使计划必须在每个成员的预测下都行得通（PETS，24.2 节），只做很短的模型滚动（MBPO，同一节），以及用新控制器采到的数据不断重训模型、每步重新规划（24.2 节的数据循环，24.4 节的 MPC）。

The "pump energy" trajectory shows the biggest headache of model-based methods: a planner goes looking for exactly such trajectories. If the goal is to swing the link up, the optimizer searches the model for the actions that raise the energy fastest, and in the high-energy region the model has no data and guesses; it may even find swinging up "easier" than the real system does. This is called model exploitation. The usual remedies come later in this chapter: propagate trajectory samples through the ensemble's members and average their costs, so that a plan has to work under every member's prediction (PETS, Lesson 24.2), keep model rollouts short (MBPO, same lesson), and keep retraining the model on data from the new controller while replanning at every step (the data loops of Lesson 24.2, the MPC of Lesson 24.4).

一步损失只问“从真实状态出发，下一步预测得准不准”，可使用时模型的输入是它自己的预测。一个自然的修正是把训练也做成滚动：从记录的状态 \(s_t\) 出发，用记录的动作滚 k 步，每一步都和记录的状态比较，

A one-step loss only asks "starting from the true state, is the next step right?", while in use the model's input is its own prediction. A natural fix is to make training a rollout too: start from a recorded state \(s_t\), roll k steps through the recorded actions, and compare every step with the recorded states,

其中 \(\|\cdot\|_{\sigma_\Delta}\) 表示先逐分量除以 \(\sigma_\Delta\) 再取范数。梯度要穿过整串滚动，这和 17.2 节的随时间反向传播（BPTT）完全一样：模型就是一个以 \(a_t\) 为输入、以状态为隐藏状态的 RNN。k = 1 时它就是一步损失。代价是每个样本要做 k 次前向和反向。下图在带传感器噪声的数据上比较两者：编码器读数有 0.02 rad 的噪声，角速度由差分估计，噪声 0.2 rad/s，这在真实机器人上很常见。

Here \(\|\cdot\|_{\sigma_\Delta}\) means divide each component by \(\sigma_\Delta\) before taking the norm. The gradient has to pass through the whole rollout, exactly the backpropagation through time (BPTT) of Lesson 17.2: the model is an RNN with \(a_t\) as input and the state as its hidden state. With k = 1 it is the one-step loss. The price is k forward and backward passes per sample. The figure below compares the two on data with sensor noise: the encoder reading has 0.02 rad of noise and the angular velocity, estimated by differencing, 0.2 rad/s, which is common on real robots.

为什么噪声让多步损失占优？一步损失把带噪声的 \(s_t\) 当作输入、带噪声的 \(s_{t+1}\) 当作目标；输入里的噪声会让回归系数系统性地偏小（统计里的“变量含误差”偏差），网络学到的动力学偏“迟钝”，这个偏差每一步都同向，滚动时就累积起来。k 步损失只在起点用一次带噪声的输入，后面的输入都是模型自己的平滑预测，偏差小得多。数据干净时，一步损失已经找到了好模型，多步损失只是多了一份更难的优化。所以多步损失不是免费的午餐：它是在“模型必然有误差”时，把误差往对滚动更无害的方向推。

Why does noise favour the multi-step loss? The one-step loss takes the noisy \(s_t\) as input and the noisy \(s_{t+1}\) as target; noise in the inputs systematically shrinks regression coefficients (the "errors-in-variables" bias of statistics), so the learned dynamics are too "sluggish", and since this bias points the same way at every step, it accumulates in a rollout. The k-step loss uses a noisy input only once, at the start; afterwards the inputs are the model's own smooth predictions, and the bias is much smaller. With clean data the one-step loss already finds a good model, and the multi-step loss only adds a harder optimization. So the multi-step loss is no free lunch: when the model is bound to be wrong, it pushes the errors in a direction that hurts rollouts less.

我们其实不是对这个机械臂一无所知：CAD 图纸给出了连杆的质量和长度，只是没算夹爪上的负载，也没有摩擦。与其让网络从零学起，不如让它只学物理模型漏掉的那部分，即残差：

We are not actually ignorant about this arm: the CAD drawing gives the link's mass and length; it just leaves out the payload in the gripper, and friction. Rather than have a network learn everything from scratch, let it learn only what the physics model misses, the residual:

这里的名义模型 \(f_{\mathrm{phys}}\) 把惯量和重力矩都低估了 20%（\(I=0.2\)、\(mgl=3.924\)），没有摩擦，用 5 ms 的半隐式欧拉法积分一个控制周期。下图比较“黑箱”（从零学 Δs）和“残差”两种模型，看训练数据从 20 个转移加到 1000 个时，1 s（20 步）开环滚动的误差怎样变化；测试分为与训练数据同一范围的运动，以及从更大摆角、更高速度出发（\(|\theta_0|<2.8\)、\(|\omega_0|<6\)）的运动。

The nominal model \(f_{\mathrm{phys}}\) here underestimates both inertia and gravity torque by 20% (\(I=0.2\), \(mgl=3.924\)), has no friction, and integrates one control period with 5 ms semi-implicit Euler steps. The figure below compares a "black-box" model (learning Δs from scratch) with a "residual" model as the training data grow from 20 to 1000 transitions, measuring the error of a 1 s (20-step) open-loop rollout; the tests are motions in the same range as the training data, and motions starting at larger angles and higher speeds (\(|\theta_0|<2.8\), \(|\omega_0|<6\)).

残差模型在大范围上也不完美（网络在那里同样是外推），但它出错时退回的是“略有偏差的物理”，而黑箱退回的是“随便什么”。这就是把已知结构放进模型的价值，也解释了为什么实践中常常先做参数辨识（8.3 节：动力学对惯性参数是线性的，可以用最小二乘估计），再用网络学剩下的摩擦（8.4 节）、柔性和气动这些难以建模的部分。

The residual model is not perfect on the wide range either (its network is extrapolating there too), but when it goes wrong it falls back on "slightly biased physics", whereas the black box falls back on "anything at all". That is the value of putting known structure into the model, and it is why practice often first identifies parameters (Lesson 8.3: the dynamics are linear in the inertial parameters, so least squares estimates them) and then lets a network learn what is left: friction (Lesson 8.4), flexibility, aerodynamics and other things that are hard to model.

到目前为止状态 \((\theta,\omega)\) 都是直接测到的。如果机器人只有一个摄像头呢？图像有几百上千个像素，而且一帧图像里看不到速度。做法是学一个编码器把最近两帧压缩成一个小的隐状态 \(\hl{latent}{z}\)，在隐空间里学动力学，再用解码器把预测的 \(z\) 变回图像来和真实画面比较（编码器、解码器见 18.2 节）：

So far the state \((\theta,\omega)\) was measured directly. What if the robot only has a camera? An image has hundreds or thousands of pixels, and velocity cannot be seen in a single frame. The approach is to learn an encoder that compresses the last two frames into a small latent state \(\hl{latent}{z}\), learn the dynamics in the latent space, and use a decoder to turn predicted \(z\) back into images to compare with the real frames (encoders and decoders are in Lesson 18.2):

下图的“相机”是 10×10 像素的灰度图，画的是摆杆。编码器 [200, 32, 3]，隐动力学 [4, 32, 3]，解码器 [3, 32, 100]，都是 tanh 网络（解码器输出层为 sigmoid），三者一起用上面的损失训练。注意没有任何地方告诉网络角度或角速度是什么。

The "camera" in the figure below gives 10×10-pixel grayscale images of the link. Encoder [200, 32, 3], latent dynamics [4, 32, 3], decoder [3, 32, 100], all tanh networks (sigmoid output layer for the decoder), trained together with the loss above. Note that nowhere is the network told what an angle or an angular velocity is.

这是一个很小的“世界模型”。它也说明了两点：隐状态只有在多步预测的压力下才会变成好用的状态；即使 k = 8，滚动到 20 步时误差也回升到基线附近（约 0.04），超过训练时的预测长度后模型照样会漂。真正的世界模型（PlaNet 和 Dreamer 的 RSSM、TD-MPC 的无解码器隐模型）在此基础上加了随机隐变量、循环网络和奖励预测，见 24.3 节。

This is a very small "world model". It shows two things: a latent state becomes a usable state only under the pressure of multi-step prediction; and even with k = 8 the error climbs back to near the baseline by step 20 (about 0.04): past the prediction length it was trained for, the model drifts all the same. Real world models (the RSSM of PlaNet and Dreamer, the decoder-free latent model of TD-MPC) add stochastic latents, recurrent networks and reward prediction on top of this; see Lesson 24.3.

提示：公式里带底色的符号可以悬停：\(f_\theta\) 加粗 FIG 1 的琥珀色箭头，\(\Delta s\) 加粗 FIG 2 的蓝、绿两条线，\(\sigma_{\mathrm{ens}}\) 加粗 FIG 3 的紫色标准差曲线，\(L_k\) 那一项加粗 FIG 4 的琥珀色曲线，\(r_\theta\) 加粗 FIG 5 的绿色曲线，\(z\) 给 FIG 6 的预测画面加上琥珀色边框。

Tip: the highlighted symbols in the formulas can be hovered: \(f_\theta\) thickens FIG 1's amber arrows, \(\Delta s\) thickens FIG 2's blue and green lines, \(\sigma_{\mathrm{ens}}\) thickens FIG 3's purple standard-deviation curve, the \(L_k\) term thickens FIG 4's amber curve, \(r_\theta\) thickens FIG 5's green curves, and \(z\) puts amber frames around FIG 6's predicted images.

把转移写成概率模型 \(p_\theta(s'\mid s,a)\)，这正是 20.1 节 MDP 里的 \(P(s'\mid s,a)\)。假设 \(s'=f_\theta(s,a)+\varepsilon\)，\(\varepsilon\sim\mathcal N(0,\Sigma)\)，则数据的负对数似然是

Write the transition as a probabilistic model \(p_\theta(s'\mid s,a)\), which is exactly the \(P(s'\mid s,a)\) of the MDP in Lesson 20.1. Assuming \(s'=f_\theta(s,a)+\varepsilon\) with \(\varepsilon\sim\mathcal N(0,\Sigma)\), the negative log-likelihood of the data is

\(\Sigma\) 固定为对角阵 \(\operatorname{diag}(\sigma_\Delta^2)\) 时，这正好是归一化的均方误差（16.2 节“损失 = 负对数似然”）。所以“对输出做归一化”其实是在声明“每一维的误差按它典型变化的大小来衡量”。如果让网络同时输出 \(\Sigma_\theta(s,a)\)（异方差，21.3 节 FIG 2），模型就能说“在这里我的预测本身就更不确定”，PETS 的概率集成就是这样做的（24.2 节）。

With \(\Sigma\) fixed to the diagonal \(\operatorname{diag}(\sigma_\Delta^2)\) this is exactly the normalized mean squared error ("losses are negative log-likelihoods", Lesson 16.2). So "normalizing the outputs" really declares "measure each dimension's error against the size of its typical change". If the network also outputs \(\Sigma_\theta(s,a)\) (heteroscedastic, FIG 2 of Lesson 21.3), the model can say "my prediction is inherently less certain here", which is what the probabilistic ensembles of PETS do (Lesson 24.2).

连续系统 \(\dot s=F(s,a)\) 在一个周期上的精确离散化是 \(s_{t+1}=s_t+\int_0^{\Delta t}F\,dt=s_t+\Delta t\,F(s_t,a_t)+O(\Delta t^2)\)（13.2 节的离散化）。所以 \(\Delta s\) 的大小正比于 \(\Delta t\)，而 \(s_{t+1}\) 本身的大小与 \(\Delta t\) 无关。直接预测 \(s_{t+1}\) 时，绝对误差 \(\epsilon\) 相对于变化量的比例是 \(\epsilon/(\Delta t\,|F|)\)：控制频率翻倍，同样的网络误差在“变化量”上就翻倍。预测 \(\Delta s\) 就是让网络学 \(\Delta t\,F\)，和欧拉积分、和残差网络（ResNet 的 \(x+g(x)\)）、和神经 ODE 是同一个结构。

FIG 1 的线性模型 \(\Delta s=W[s;a;1]\) 则是 13.2 节线性化的数据版：\(A_d=I+W_{:,1:2}\)，\(B_d=W_{:,3}\)。用最小二乘从数据拟合 \(A_d,B_d\) 是经典的系统辨识，在工作点附近往往已经够 LQR 用；它的局限正是 FIG 1 里大角度处 sin θ 的弯曲。

The exact discretization of a continuous system \(\dot s=F(s,a)\) over one period is \(s_{t+1}=s_t+\int_0^{\Delta t}F\,dt=s_t+\Delta t\,F(s_t,a_t)+O(\Delta t^2)\) (discretization in Lesson 13.2). So \(\Delta s\) scales with \(\Delta t\) while \(s_{t+1}\) itself does not. When predicting \(s_{t+1}\) directly, an absolute error \(\epsilon\) is a fraction \(\epsilon/(\Delta t\,|F|)\) of the change: double the control rate and the same network error doubles relative to the change. Predicting \(\Delta s\) makes the network learn \(\Delta t\,F\), the same structure as Euler integration, as a residual network (ResNet's \(x+g(x)\)), and as a neural ODE.

FIG 1's linear model \(\Delta s=W[s;a;1]\) is the data-driven version of Lesson 13.2's linearization: \(A_d=I+W_{:,1:2}\), \(B_d=W_{:,3}\). Fitting \(A_d,B_d\) from data by least squares is classical system identification, often good enough for LQR near an operating point; its limit is exactly the bending of sin θ at large angles in FIG 1.

记 \(e_t=\|\hat s_t-s_t\|\)，真实的下一步是 \(s_{t+1}=f(s_t,a_t)\)。若模型在滚动所到的每个（预测）状态上的一步误差不超过 \(\varepsilon\)，且真实动力学对状态是 \(\Lambda\)-Lipschitz 的，那么

Let \(e_t=\|\hat s_t-s_t\|\) with the true next step \(s_{t+1}=f(s_t,a_t)\). If the model's one-step error is at most \(\varepsilon\) at every (predicted) state the rollout visits, and the true dynamics are \(\Lambda\)-Lipschitz in the state, then

\(\Lambda<1\)（收缩、强阻尼）时误差有界，约为 \(\varepsilon/(1-\Lambda)\)；\(\Lambda=1\) 时线性增长 \(H\varepsilon\)；\(\Lambda>1\)（不稳定或混沌）时指数增长。几乎无阻尼的摆接近 \(\Lambda\approx1\)：小的频率误差变成随时间线性增长的相位误差。更糟的是前提“所到之处 \(\varepsilon\) 都小”：预测状态一旦离开数据，\(\varepsilon\) 本身就变大（FIG 3）。这和行为克隆的复合误差（22.1 节）是同一种机制：训练时的输入来自数据分布，使用时的输入来自模型自己。

With \(\Lambda<1\) (contracting, heavily damped) the error stays bounded, about \(\varepsilon/(1-\Lambda)\); with \(\Lambda=1\) it grows linearly as \(H\varepsilon\); with \(\Lambda>1\) (unstable or chaotic) it grows exponentially. A nearly undamped pendulum is close to \(\Lambda\approx1\): a small frequency error becomes a phase error growing linearly with time. Worse is the premise "\(\varepsilon\) is small wherever it goes": once the predicted state leaves the data, \(\varepsilon\) itself grows (FIG 3). This is the same mechanism as the compounding error of behaviour cloning (Lesson 22.1): at training time the inputs come from the data distribution, at use time from the model itself.

对 \(L_k\) 求导时，第 \(j\) 步的误差通过所有更早的步影响参数。记状态雅可比 \(J_i=\partial f_\theta/\partial s\big|_{(\hat s_i,a_i)}\)（连乘里时间越晚的雅可比越靠左）：

Differentiating \(L_k\), the error at step \(j\) reaches the parameters through every earlier step. With the state Jacobians \(J_i=\partial f_\theta/\partial s\big|_{(\hat s_i,a_i)}\) (in the product, later Jacobians stand to the left):

雅可比的连乘和 17.2 节 RNN 里的一样：\(\|J\|>1\) 时梯度爆炸，\(<1\) 时消失，所以 k 不能太大，常用梯度裁剪；FIG 4 和 FIG 6 的代码里，\(\partial L/\partial\hat s\) 正是用 nn.js 的 net.backward 返回的输入梯度一步步往回传的。另一种做法是 DaD（Data as Demonstrator，Venkatraman 等 2015）：先滚动，再把“从预测状态回到真实轨迹”的修正作为新的一步训练样本加入数据，就像 DAgger 之于行为克隆。

The product of Jacobians is the same as in Lesson 17.2's RNNs: with \(\|J\|>1\) gradients explode, with \(<1\) they vanish, so k cannot be too large and gradient clipping is common; in the code of FIG 4 and FIG 6, \(\partial L/\partial\hat s\) is passed back step by step using the input gradients returned by nn.js's net.backward. An alternative is DaD (Data as Demonstrator, Venkatraman et al. 2015): roll out first, then add "the correction from the predicted state back to the true trajectory" to the data as new one-step training samples, much as DAgger does for behaviour cloning.

21.3 节的分解直接适用：\(K\) 个成员，各自输出 \(\mu_k(s,a)\)（以及可选的 \(\Sigma_k\)），

The decomposition of Lesson 21.3 applies directly: \(K\) members, each outputting \(\mu_k(s,a)\) (and optionally \(\Sigma_k\)),

第一项是偶然部分（过程噪声、没观测到的因素），第二项是认知部分（数据不够）。滚动时有两种传播方式：每个成员各滚各的（FIG 3 的做法），或者每一步随机重挑一个成员；PETS 的“轨迹采样”两种都考察过（24.2 节）。无论哪种，FIG 3 已经提醒过：成员共享结构和数据，在远离数据处往往一起错，分歧只是认知不确定性的一个偏乐观的估计。

The first term is the aleatoric part (process noise, unobserved factors), the second the epistemic part (not enough data). In a rollout it can be propagated in two ways: each member rolls on its own (as in FIG 3), or a member is re-drawn at random at every step; the "trajectory sampling" of PETS studies both (Lesson 24.2). Either way, FIG 3 has already warned: the members share architecture and data, often go wrong together far from the data, and their disagreement is an optimistic estimate of the epistemic uncertainty.

残差模型可以写在不同层次上：加在下一状态上（FIG 5 的 \(f_{\mathrm{phys}}+r_\theta\)），或加在力矩上，\(M(\mathbf q)\ddot{\mathbf q}+\mathbf b(\mathbf q,\dot{\mathbf q})=\boldsymbol\tau+\boldsymbol\tau_\theta(\mathbf q,\dot{\mathbf q},\boldsymbol\tau)\)，后者让网络只补“未建模的力”，再交给积分器。物理参数 \(\varphi\) 也可以和 \(\theta\) 一起拟合，但要小心两者“抢”同一部分误差，常见做法是先辨识 \(\varphi\) 再学 \(\theta\)，或者对 \(r_\theta\) 加正则让它尽量小。隐空间模型的损失（FIG 6）里，解码器的重建项防止所有 \(z\) 塌缩到同一点（18.4 节讲过的“塌缩”）；不用解码器的做法（TD-MPC）改用奖励和价值预测加上隐状态的一致性来防止塌缩，见 24.3 节。

A residual can sit at different levels: on the next state (FIG 5's \(f_{\mathrm{phys}}+r_\theta\)), or on the torque, \(M(\mathbf q)\ddot{\mathbf q}+\mathbf b(\mathbf q,\dot{\mathbf q})=\boldsymbol\tau+\boldsymbol\tau_\theta(\mathbf q,\dot{\mathbf q},\boldsymbol\tau)\), where the network only supplies the "unmodelled force" and the integrator does the rest. The physical parameters \(\varphi\) can also be fitted together with \(\theta\), but beware of the two "competing" for the same part of the error; the usual practice is to identify \(\varphi\) first and then learn \(\theta\), or to regularize \(r_\theta\) to stay small. In the latent model's loss (FIG 6) the decoder's reconstruction term keeps all \(z\) from collapsing to one point (the "collapse" of Lesson 18.4); decoder-free approaches (TD-MPC) prevent collapse with reward and value prediction plus latent consistency instead; see Lesson 24.3.

学到的模型在机器人里有三种典型用法。当仿真器用：在里面训练策略或做规划，省下真实机器人的时间（24.2、24.3 节）。放进控制器：MPC 每个周期在模型上滚动、优化动作（24.4 节），或者用模型做前馈补偿。补全仿真器：把学到的部件插进物理仿真器，缩小仿真和现实的差距（23.1 节）。数据通常就是机器人日常运行时的日志：关节编码器、电流（力矩）、IMU、相机，频率从几十 Hz 到上千 Hz。

Learned models are used in robots in three typical ways. As a simulator: train policies or plan inside it and save time on the real robot (Lessons 24.2, 24.3). Inside a controller: MPC rolls the model out and optimizes actions every period (Lesson 24.4), or the model supplies feedforward compensation. To complete a simulator: plug learned parts into a physics simulator to narrow the gap between simulation and reality (Lesson 23.1). The data are usually the robot's everyday logs: joint encoders, currents (torques), IMU, cameras, at rates from tens of Hz to over a thousand.