Part VI · 第 23 章机器人强化学习 · 23.1Part VI · Ch. 23 Robot Reinforcement Learning · 23.1

机器人强化学习与 sim-to-realRobot RL & Sim-to-Real

强化学习在机器人上最成功的地方，是腿式运动和灵巧手：先在仿真里让成千上万个机器人并行试错，再把学到的策略搬到真机上。本节把这条流水线拆开来逐段实验：优化器怎样钻仿真器的空子；PPO 在许多并行环境里实时训练一个关节，动作接口选 PD 目标角还是力矩；奖励怎样写、课程怎样排；只在仿真里才有的“特权信息”怎样通过非对称 actor-critic 和教师–学生蒸馏用起来；现实差距（延迟、执行器、传感器）从哪来，系统辨识和执行器网络怎样把它缩小；最后核对 ANYmal、OpenAI 灵巧手和腿式跑酷这些里程碑到底做了什么。 Reinforcement learning has succeeded on robots above all in legged locomotion and dexterous hands, by the same recipe: let thousands of simulated robots try things in parallel, then move the learned policy to the real machine. This lesson takes that pipeline apart and experiments with each piece: how an optimizer exploits the simulator; PPO training a joint live over many parallel environments, with PD targets or torques as the action; how to write rewards and order a curriculum; how "privileged information" that exists only in simulation is put to use through asymmetric actor-critic and teacher–student distillation; where the reality gap comes from (latency, actuators, sensing) and how system identification and actuator networks shrink it; and finally what the landmark results on ANYmal, OpenAI's dexterous hand and legged parkour actually did.

约 55 分钟~55 min 前置：20.4 Actor-Critic：PPO 与 SAC · 20.3 策略梯度 · 22.1 模仿学习：BC 与 DAgger · 9.4 执行器与传感器 · 之后 23.2 域随机化Prereqs: 20.4 Actor-Critic: PPO & SAC · 20.3 Policy gradients · 22.1 Imitation learning: BC & DAgger · 9.4 Actuators & sensors · next 23.2 Domain randomization 5 个交互5 interactives

1为什么机器人需要它Why robots need this

有两类机器人问题，经典控制做得很辛苦：腿式运动（四条腿或两条腿在碎石、台阶、泥地上走、跑、摔倒后爬起来）和灵巧手操作（五指手在手里翻转一个方块）。它们的共同点是接触多、接触模式不断切换，模型写得出来（8.4 节）却很难拿来在线求解；而“好”的行为又很容易打分：走得快不快、摔没摔、方块转到目标朝向没有。这正是强化学习的地盘：只需要奖励，不需要专家演示，也不需要把控制律写出来。

代价是样本。PPO（20.4 节）这类 on-policy 方法训练一个运动策略要上亿个环境步，按常见的 50 Hz 控制频率，$10^8$ 步相当于一台真机不停地走约 23 天，还要摔几千次；OpenAI 的灵巧手更是用了约 100 年的仿真经验（第 6 节）。所以今天的主流做法是：在仿真里训练，用 GPU 同时跑几千个机器人，几分钟到几小时练完；然后零样本部署到真机（sim-to-real）。这把问题变成了三件工程：让仿真足够快（并行仿真）、让学习问题足够好学（观测、动作、奖励、课程的设计）、让仿真和真机足够像，或者让策略对“不像”足够不敏感（现实差距、系统辨识、23.2 节的域随机化）。

本节的贯穿例子是一个受重力的机器人关节：它可以看成腿上的一个关节或机械臂的肩关节，要按指令转到目标角并保持住。它小到能在你的浏览器里几秒钟训练完，又足够真实：有力矩上限、控制周期、延迟、电机滞后和传感器噪声，sim-to-real 里的每个问题都能在它身上看到。

Two kinds of robot problems are hard work for classical control: legged locomotion (four or two legs walking, running and getting back up on gravel, stairs and mud) and dexterous manipulation (a five-fingered hand turning a block over in its palm). Both involve many contacts with constantly switching contact modes; the model can be written down (Lesson 8.4) but is hard to solve online, while "good" behaviour is easy to score: how fast it walked, whether it fell, whether the block reached its target orientation. That is reinforcement learning's home ground: it needs only a reward, no expert demonstrations and no control law written by hand.

The price is samples. An on-policy method such as PPO (Lesson 20.4) needs on the order of a hundred million environment steps to train a locomotion policy, which at a typical 50 Hz control rate is about 23 days of non-stop walking on one real robot, with thousands of falls; OpenAI's dexterous hand used about 100 years of simulated experience (Section 6). So the mainstream recipe today is to train in simulation, with a GPU running thousands of robots at once, in minutes to hours, and then deploy zero-shot on the real robot (sim-to-real). That turns the problem into three pieces of engineering: make the simulation fast (parallel simulation), make the learning problem easy to learn (the design of observations, actions, rewards and curricula), and make simulation and reality alike, or make the policy insensitive to their differences (the reality gap, system identification and the domain randomization of Lesson 23.2).

The running example is one robot joint under gravity: think of it as a joint of a leg or the shoulder of an arm, commanded to turn to a target angle and hold it. It is small enough to train in your browser in seconds, yet realistic enough to show every sim-to-real problem: a torque limit, a control period, latency, motor lag and sensor noise.

2直觉：优化器会钻仿真器的空子Intuition: the optimizer exploits the simulator

先不用神经网络，只看一个两参数的“策略”：水平放置（不受重力）的关节，从 −0.3 rad 转回 0，控制器是 $\tau=-k_pq-k_d\dot q$，每 40 ms（25 Hz）更新一次力矩，中间保持不变，力矩上限 12 N·m。代价 $J$ 是 2 秒内角度误差平方的平均（加一点力矩惩罚），除以“什么都不做”时的值：$J=1$ 等于没控制，越小越好。仿真器里没有延迟、电机理想；真机有通信延迟、电机滞后（力矩跟不上指令）和偏弱的电机。

强化学习、CEM（6.11 节）或任何优化器在仿真里做的事都一样：找使 $\hl{cem}{J_{\text{sim}}}$ 最小的参数。下图左边是仿真里的代价地形，右边是同一组增益在真机上的代价地形。按“在仿真里优化”，CEM 会一路爬到很高的增益：在仿真里，增益越高收敛越快，它没有理由停下。可这片区域恰好挨着真机的不稳定区。

Forget neural networks for a moment and look at a two-parameter "policy": a horizontal joint (no gravity) must turn from −0.3 rad back to 0 under the controller $\tau=-k_pq-k_d\dot q$, with the torque updated every 40 ms (25 Hz), held in between, and limited to 12 N·m. The cost $J$ is the mean squared angle error over 2 seconds (plus a little torque penalty), divided by its value when doing nothing: $J=1$ means no control at all, and lower is better. The simulator has no latency and an ideal motor; the robot has communication latency, motor lag (the torque cannot follow the command instantly) and a weaker motor.

Reinforcement learning, CEM (Lesson 6.11) or any other optimizer does the same thing in simulation: find the parameters that minimize $\hl{cem}{J_{\text{sim}}}$. On the left below is the cost landscape in simulation, on the right the landscape of the same gains on the robot. Press "Optimize in sim" and CEM climbs to high gains: in simulation, higher gains converge faster and there is no reason to stop. But that region lies right next to the robot's unstable region.

FIG 1同一组增益，两张代价地形：仿真的最优点在真机上很差The same gains, two cost landscapes: the simulator's optimum is poor on the robot

两张图横轴 $k_p$（0–200），纵轴 $k_d$（0–40），背景是代价 $J$，越青 = 代价越低（8 档，对数刻度，从 0.07 到 1）；红色格子表示 $J\ge1$：比不控制还差，多半是振荡或发散。左边是仿真器（无延迟、理想电机），右边是真机（滑块设定的延迟、电机滞后和强度）。琥珀色圆点是当前增益，在任一张图上都可以拖动。按钮运行 CEM（每轮 40 个样本、保留最好的 8 个、15 轮），走过的路径在“仿真里优化”时画成紫色，“在一组随机化的仿真里优化”时画成绿色；后者最小化 10 个仿真器的平均代价，这些仿真器的延迟取 0–80 ms、电机滞后取 0–40 ms、强度取 0.7 或 1（这就是 23.2 节的域随机化）。下方：当前增益下关节角 $q(t)$ 的响应，紫线是仿真，琥珀线是真机。 Both maps have $k_p$ across (0–200) and $k_d$ up (0–40); the background is the cost $J$, more cyan = lower cost (8 bands on a log scale from 0.07 to 1); red cells mean $J\ge1$, worse than no control, usually an oscillation or divergence. Left: the simulator (no latency, ideal motor). Right: the robot (latency, motor lag and strength set by the sliders). The amber dot is the current pair of gains; drag it on either map. The buttons run CEM (40 samples per round, keep the best 8, 15 rounds); its path is drawn in purple for "Optimize in sim" and in green for "Optimize over randomized sims", which minimizes the average cost over 10 simulators with latency 0–80 ms, motor lag 0–40 ms and strength 0.7 or 1 (the domain randomization of Lesson 23.2). Below: the response $q(t)$ at the current gains, the purple line in simulation and the amber line on the robot.

试一试Try this

按“在仿真里优化”：CEM 停在 $(k_p,k_d)\approx(76,12)$，$J_{\text{sim}}\approx0.074$。默认真机（延迟 40 ms）上同一组增益的代价是 0.127，已经比仿真差 1.7 倍；把延迟拉到 60 ms 以上，真机代价超过仿真代价的 2 倍（60 ms 时约 0.24，80 ms 时约 0.47），右图里红色的不稳定区几乎贴到了琥珀点上。Press "Optimize in sim": CEM stops near $(k_p,k_d)\approx(76,12)$ with $J_{\text{sim}}\approx0.074$. On the default robot (40 ms latency) the same gains cost 0.127, already 1.7 times the simulated cost; push the latency to 60 ms or more and the robot's cost exceeds twice the simulated one (about 0.24 at 60 ms, 0.47 at 80 ms), with the red unstable region on the right almost touching the amber dot.
保持延迟 $\ge60$ ms，找一组在两边都过得去的增益：真机代价 $<0.13$，同时仿真代价 $<0.11$。可以自己拖，也可以按“在一组随机化的仿真里优化”：它停在约 $(28,8)$，仿真里差一点（0.099），但真机上 60 ms 时 0.124、80 ms 时也只有 0.138。Keep the latency at 60 ms or more and find gains that are acceptable on both sides: a robot cost below 0.13 and a simulated cost below 0.11. Drag the dot yourself, or press "Optimize over randomized sims": it stops near $(28,8)$, slightly worse in simulation (0.099) but 0.124 on the robot at 60 ms and still only 0.138 at 80 ms.

核心洞见Key insight

优化器不知道哪部分仿真是真的、哪部分只是近似，它会利用一切能让代价变小的东西，包括仿真的缺陷：这里是“没有延迟”，在真实的接触仿真里可能是穿模、过大的摩擦、无限快的电机。仿真越不准，优化越彻底，部署后的落差往往越大。sim-to-real 的所有手段都在回答同一个问题：怎样让 $J_{\text{sim}}$ 的最优点在真机上也好。三条路：把仿真做准（系统辨识，第 4 节），让策略不依赖不准的部分（域随机化、动作接口的选择），以及让策略在线识别它所处的环境（历史编码器与教师–学生，第 4 节和 23.2 节）。

The optimizer does not know which parts of the simulator are real and which are approximations; it uses anything that lowers the cost, flaws included: here "no latency", in a real contact simulator perhaps interpenetration, too much friction, infinitely fast motors. The less accurate the simulator and the more thorough the optimization, the larger the drop after deployment tends to be. Every sim-to-real technique answers one question: how to make the optimum of $J_{\text{sim}}$ good on the robot too. Three routes: make the simulator accurate (system identification, Section 4), make the policy independent of the inaccurate parts (domain randomization, the choice of action interface), and let the policy identify its environment online (history encoders and teacher–student training, Section 4 and Lesson 23.2).

3动手：在许多并行环境里用 PPO 训练一个关节Play: train a joint with PPO over many parallel environments

现在换成真正的强化学习。关节竖直放置，受重力（$q=0$ 为自然下垂），每个回合开始时随机给一个指令角 $q^\star\in[-1.5,1.5]$ rad，2 秒内要转过去并保持住。物理仿真 200 Hz，策略 25 Hz：每个动作保持 8 个物理步，这与腿式机器人的常见设置同构（仿真跑得比策略快几倍）。策略的输入是角度误差 $q-q^\star$、角速度、指令和上一步的动作；输出一个高斯分布的均值。奖励是每步一个跟踪项 $\exp(-(q-q^\star)^2/0.05)$，减去很小的力矩惩罚和动作变化率惩罚。

算法就是 20.4 节的 PPO：每次迭代，$N$ 个环境用当前策略同时各跑一个回合（50 步），用 GAE 算优势，再做 4 轮、每轮 4 个小批量的截断更新。并行环境越多，每次迭代的数据越多，梯度越准，但每次迭代要模拟的步数也越多。在 GPU 仿真器上，几千个环境的一步和一个环境的一步花的时间相差不大，所以“迭代次数”近似就是墙钟时间；在 CPU 上逐个仿真，“环境步数”才是。

还有一个设计选择：动作接口。“直接力矩”让策略每 40 ms 给一个力矩；“PD 目标角”让策略给一个目标角 $\hl{pd}{q^{\text{tgt}}=a}$，由电机驱动器里 200 Hz 的 PD 回路 $\tau=K_p(q^{\text{tgt}}-q)-K_d\dot q$（$K_p=20,\ K_d=1.5$）去跟踪。腿式机器人的学习控制器大多用后者。右下角把训练出来的策略放到“真机”上：延迟 30 ms、电机滞后 20 ms、强度 0.85、库仑摩擦和传感器噪声。

Now for real reinforcement learning. The joint is vertical, under gravity ($q=0$ hangs straight down); each episode draws a random command $q^\star\in[-1.5,1.5]$ rad, and the joint has 2 seconds to get there and hold it. Physics runs at 200 Hz and the policy at 25 Hz, each action held for 8 physics steps, the same structure as common legged-robot setups (the simulator runs several times faster than the policy). The policy sees the angle error $q-q^\star$, the joint speed, the command and its previous action, and outputs the mean of a Gaussian. The reward per step is a tracking term $\exp(-(q-q^\star)^2/0.05)$ minus small penalties on torque and on the action rate.

The algorithm is the PPO of Lesson 20.4: in every iteration, $N$ environments each run one episode (50 steps) at the same time with the current policy, GAE computes the advantages, and then 4 epochs of 4 clipped minibatch updates follow. More parallel environments give more data per iteration and a more accurate gradient, but also more steps to simulate per iteration. On a GPU simulator, one step of thousands of environments takes about as long as one step of a single one, so the number of iterations is roughly the wall-clock time; on a CPU that simulates them one by one, the number of environment steps is.

One more design choice: the action interface. "Torque" lets the policy output a torque every 40 ms; "PD target" lets it output a target angle $\hl{pd}{q^{\text{tgt}}=a}$, which a 200 Hz PD loop in the motor drive, $\tau=K_p(q^{\text{tgt}}-q)-K_d\dot q$ ($K_p=20,\ K_d=1.5$), then tracks. Most learned legged controllers use the latter. The bottom-right panel deploys the trained policy on the "robot": 30 ms latency, 20 ms motor lag, strength 0.85, Coulomb friction and sensor noise.

FIG 2实时训练：PPO 驱动一批并行的仿真关节，再部署到“真机”Live training: PPO drives a batch of parallel simulated joints, then deploys to the "robot"

左：前 16 个并行环境，回放上一次迭代里它们各自的回合（带探索噪声）；紫色虚线是各自的指令角，连杆误差小于 0.1 rad 时画成绿色，否则琥珀色。环境数少于 16 时，空着的格子画成半透明的静止连杆。右上：测试误差（均值策略在 8 个固定指令上最后 0.5 秒的平均 $|q-q^\star|$，对数刻度）随迭代次数（琥珀线），重置后上一次的曲线保留为紫色虚线，绿色虚线是 0.05 rad。右下：指令 1 rad 时均值策略的响应，青线是仿真，琥珀线是真机，紫色虚线是指令。读数记录每种环境数第一次达到 0.1 rad 时用了多少次迭代、多少环境步，以及最后一个训练到 0.05 rad 以下的 PD 策略和力矩策略在真机上的误差。训练每帧最多花约 7 ms；随机数有固定种子，同一设置每次结果相同。 Left: the first 16 parallel environments, replaying their episodes from the last iteration (with exploration noise); the purple dashed line is each one's command, and a link is drawn green when it is within 0.1 rad of it, amber otherwise. With fewer than 16 environments, the unused cells show a translucent, idle link. Top right: the test error (mean $|q-q^\star|$ over the last 0.5 s of the mean policy on 8 fixed commands, log scale) against iterations (amber line); after a reset, the previous curve stays as a purple dashed line, and the green dashed line is 0.05 rad. Bottom right: the response of the mean policy to a 1 rad command, the cyan line in simulation and the amber line on the robot, with the command as a purple dashed line. The readout records, for each number of environments, the iterations and environment steps it took to first reach 0.1 rad, and the robot error of the latest PD and torque policies trained below 0.05 rad. Training spends at most about 7 ms per frame; the random numbers are seeded, so the same settings always train the same way.

默认设置（PD 目标角，16 个环境）按“训练”：一开始测试误差约 0.85 rad（连杆基本不动），第 30 次迭代降到 0.1 rad 以下，第 37 次降到 0.05 rad 以下。With the defaults (PD target, 16 environments) press "Train": the test error starts around 0.85 rad (the link barely moves), drops below 0.1 rad at iteration 30 and below 0.05 rad at iteration 37.
分别用 4 个和 64 个环境（PD）训练到 0.1 rad 以下（切换会重置）：4 个环境要 35 次迭代、只用 7000 个环境步；64 个环境只要 25 次迭代，却用了 80000 步。读数里 64 的迭代数比 4 少时完成。再试 128：27 次迭代、172800 步，迭代次数不再下降。Train to below 0.1 rad with 4 and with 64 environments (PD; switching resets): 4 environments need 35 iterations but only 7000 environment steps; 64 need only 25 iterations but 80000 steps. The task completes when the readout shows fewer iterations for 64 than for 4. Then try 128: 27 iterations and 172800 steps; the iteration count stops falling.
16 个环境，先用 PD 目标角训练到 0.05 rad 以下，再切到“直接力矩”训练到 0.05 rad 以下（力矩接口在仿真里反而更快：23 次迭代就到 0.1 rad）。在默认真机上两者都还行（约 0.045 和 0.069 rad）；把测试用真机的电机强度降到 0.6 以下：PD 策略的真机误差仍在 0.05 rad 左右（低于 0.07 即可），力矩策略是它的 2 倍以上（刚训练好时约 0.22 rad，多训练 20 次迭代后约 0.11）。With 16 environments, first train the PD-target policy below 0.05 rad, then switch to "torque" and train it below 0.05 rad too (in simulation the torque interface is actually faster: 0.1 rad after 23 iterations). On the default robot both are fine (about 0.045 and 0.069 rad); now lower the test robot's motor strength to 0.6 or less: the PD policy's robot error stays around 0.05 rad (below 0.07 is enough), while the torque policy's is more than twice as large (about 0.22 rad right after training, about 0.11 after 20 more iterations).

有意思的细节Things worth noticing

(1) 环境越多，迭代越少，但“每个样本学到的东西”越少：从 4 到 64 个环境，样本多了 10 倍以上，迭代只少了约三成，到 128 个就饱和了。这正是 GPU 并行仿真的算账方式：样本几乎免费，墙钟时间才贵，所以宁可多花样本也要少迭代；在真机上则正好反过来（第 6 节的真机 RL）。 (2) PD 目标角接口的好处不在于学得快，而在于把一部分控制留在了高频、可靠的 PD 回路里：电机变弱、指令晚到 30 ms，PD 回路照样在 200 Hz 下修正误差；力矩策略则把“仿真里电机正好这么强”学进了权重。 (3) 动作 $a=0$ 对应“回到默认姿态”，这让初始的随机策略就是安全、平稳的，探索从一个合理的地方开始。

(1) More environments mean fewer iterations but less learned per sample: going from 4 to 64 environments costs more than 10 times the samples and saves only about a third of the iterations, and by 128 it saturates. That is exactly how GPU-parallel simulation keeps its books: samples are almost free and wall-clock time is expensive, so it is worth spending samples to save iterations; on a real robot it is the other way round (real-world RL in Section 6). (2) The PD-target interface does not win by learning faster but by leaving part of the control in a fast, reliable PD loop: if the motor is weaker or the command arrives 30 ms late, the PD loop still corrects the error at 200 Hz, whereas the torque policy has learned "the motor is exactly this strong in simulation" into its weights. (3) The action $a=0$ means "go back to the default posture", which makes the initial random policy safe and smooth, so exploration starts from a sensible place.

4另外三个视角：奖励与课程、特权信息、现实差距Three more views: rewards and curricula, privileged information, the reality gap

奖励怎样写，课程怎样排How to write the reward, how to order a curriculum

机器人 RL 的奖励几乎都是若干项的加权和：一两个任务项（跟踪指令速度、到达目标朝向）加上一串正则项（力矩、关节加速度、动作变化率、脚打滑、机身碰撞），20.1 节举过腿式机器人的例子。任务项的形状决定了学习信号：“到了才给分”的稀疏奖励最诚实，可在随机策略几乎碰不到目标时，它几乎没有梯度；像 $\exp(-e^2/\sigma^2)$ 这样的稠密核处处给出“离目标多近”的信号，但核的宽度 $\sigma$ 也决定了策略最终在乎多小的误差。

课程（curriculum）把任务从易排到难。下图同时训练三个策略：稠密核、稀疏奖励（误差小于 0.05 rad 才得 1 分），以及带课程的稀疏奖励：容差从 0.8 rad 开始，只要回合最后 0.5 秒里有 60% 的步落在容差内，就把容差收紧 20%，直到 0.05 rad。这是 Rudin 等（2021）“地形难度随表现升降”的课程的同一个思路，只是这里升级的是精度而不是地形高度。

Robot RL rewards are almost always a weighted sum of terms: one or two task terms (track the commanded velocity, reach the target orientation) plus a list of regularizers (torque, joint acceleration, action rate, foot slip, body collisions); Lesson 20.1 gave the legged-robot example. The shape of the task term decides the learning signal: a sparse "points only when you get there" reward is the most honest, but while a random policy hardly ever reaches the goal it carries almost no gradient; a dense kernel such as $\exp(-e^2/\sigma^2)$ says "how close" everywhere, but its width $\sigma$ also decides how small an error the policy ends up caring about.

A curriculum orders the task from easy to hard. The figure below trains three policies side by side: the dense kernel, a sparse reward (1 point only when the error is below 0.05 rad), and the sparse reward with a curriculum: the tolerance starts at 0.8 rad and shrinks by 20% whenever 60% of the steps in the last 0.5 s of the episodes fall inside it, down to 0.05 rad. It is the same idea as the curriculum of Rudin et al. (2021), in which terrain difficulty rises and falls with performance, except that here precision is promoted instead of terrain height.

FIG 3三种奖励同时训练：稠密核、稀疏奖励、带课程的稀疏奖励Three rewards trained side by side: a dense kernel, a sparse reward, a sparse reward with a curriculum

测试误差（与 FIG 2 相同的定义，对数刻度）随迭代次数：蓝线是稠密核，红线是稀疏奖励，绿线是带课程的稀疏奖励，绿色虚线是课程当前的容差。三个策略都用 PD 目标角、16 个环境、同一个种子，轮流各做一次迭代。读数里的“末段命中率”是上一次迭代中，回合最后 0.5 秒落在容差内的步数比例。 Test error (defined as in FIG 2, log scale) against iterations: the blue line is the dense kernel, the red line the sparse reward, the green line the sparse reward with a curriculum, and the green dashed line the curriculum's current tolerance. All three use PD targets, 16 environments and the same seed, and take turns doing one iteration each. The "final-step hit rate" in the readout is the fraction of steps in the last 0.5 s of the previous iteration's episodes that fell inside the tolerance.

按“训练”，等三条曲线都过了 40 次迭代：带课程的策略误差已低于 0.02 rad（约 0.013），稠密核约 0.037，纯稀疏奖励还在 0.1 rad 左右；注意纯稀疏的红线前 20 次迭代来回摆动（第 10 次约 0.6，第 20 次又回到约 0.82），之后才开始下降。再训练下去，三者在 80 次迭代时都到了 0.02 rad 以下（稠密核约 0.016，两种稀疏奖励约 0.01）：稠密核的宽度 $\sigma$ 让它不太在乎很小的误差。Press "Train" and wait until all three curves pass 40 iterations: the curriculum policy is already below 0.02 rad (about 0.013), the dense kernel at about 0.037, while the plain sparse reward is still around 0.1 rad; note that the red sparse line wanders for the first 20 iterations (about 0.6 at iteration 10, back to about 0.82 at 20) before it starts to fall. Keep training: by 80 iterations all three are below 0.02 rad (the dense kernel about 0.016, both sparse rewards about 0.01): the width $\sigma$ of the dense kernel makes it care little about very small errors.

课程也可以排在正则项上。Hwangbo 等（2019）训练 ANYmal 时发现，一开始就把力矩、关节速度这些惩罚开满，策略最容易学到的是“站着别动”这个局部最优；他们把所有正则项乘上一个课程因子 $k_c$，从 $k_0=0.3$ 开始按 $k_c\leftarrow k_c^{\,k_d}$（$0<k_d<1$）单调升到 1：先学会完成任务，再慢慢学会守规矩。

A curriculum can also be put on the regularizers. Hwangbo et al. (2019), training ANYmal, found that with the torque and joint-speed penalties at full strength from the start, the easiest thing for the policy to learn is the local optimum "stand still"; they multiplied all regularizers by a curriculum factor $k_c$ that starts at $k_0=0.3$ and rises monotonically to 1 by $k_c\leftarrow k_c^{\,k_d}$ ($0<k_d<1$): first learn to do the task, then slowly learn to behave.

特权信息：非对称 actor-critic 与教师–学生蒸馏Privileged information: asymmetric actor-critic and teacher–student distillation

仿真里一切都是已知的：真实速度、摩擦系数、负载质量、脚下的地形高度。真机只有它的传感器：编码器、IMU、有时是一台噪声很大的深度相机。部署的策略只能用真机的观测 $o_t$，可训练时没有理由不用仿真里的特权信息 $x^{\text{priv}}_t$。有两种用法。

非对称 actor-critic（Pinto 等，2018）：critic 只在训练时用，所以让它看完整状态 $V_\phi(s_t)$，actor 只看 $o_t$。critic 的任务（估计回报）因此容易得多，优势估计更准。OpenAI 的灵巧手（2018）就这样训练：价值网络能看到真机上拿不到的信息。
教师–学生（Chen 等，2019；Lee 等，2020）：先用 RL 训练一个看得见特权信息的教师策略 $\pi^{\text{T}}(a\mid o_t,x^{\text{priv}}_t)$，这是一个容易得多的 RL 问题；再用监督学习训练只看观测历史的学生 $\pi^{\text{S}}(a\mid o_{t-K:t})$ 去模仿教师。学生必须从历史里推断出教师直接看到的东西（这里是速度和负载），这一步就是 22.1 节的 DAgger：让学生自己开，教师在学生到达的状态上给标签。

下图的关节用直接力矩控制，末端挂着每回合随机的负载（0 到 1，越重越难举，惯量也越大）。教师看到干净的角度、速度和负载；学生只有一个带噪声的角度读数（标准差 0.01 rad，没有速度传感器）和最近 4 步的读数与动作；对照组是同样只有带噪声角度、但没有历史的策略，直接用 PPO 训练。

In simulation everything is known: the true velocity, the friction coefficient, the payload mass, the height of the terrain underfoot. The real robot has only its sensors: encoders, an IMU, sometimes a very noisy depth camera. The deployed policy can use only the robot's observations $o_t$, but there is no reason not to use the privileged information $x^{\text{priv}}_t$ of the simulator during training. There are two ways.

Asymmetric actor-critic (Pinto et al., 2018): the critic is used only during training, so it may see the full state, $V_\phi(s_t)$, while the actor sees only $o_t$. The critic's job (estimating returns) becomes much easier and the advantage estimates more accurate. OpenAI's dexterous hand (2018) was trained this way: its value network could see information unavailable on the real robot.
Teacher–student (Chen et al., 2019; Lee et al., 2020): first train with RL a teacher policy $\pi^{\text{T}}(a\mid o_t,x^{\text{priv}}_t)$ that sees the privileged information, a much easier RL problem; then train with supervised learning a student $\pi^{\text{S}}(a\mid o_{t-K:t})$ that sees only the history of observations and imitates the teacher. The student must infer from its history what the teacher sees directly (here the speed and the payload), and that step is the DAgger of Lesson 22.1: the student drives, and the teacher labels the states the student reaches.

In the figure below the joint is torque-controlled and carries a random payload each episode (0 to 1; heavier means harder to lift and more inertia). The teacher sees the clean angle, the speed and the payload; the student has only a noisy angle reading (standard deviation 0.01 rad, no velocity sensor) and the last 4 readings and actions; the baseline sees the same noisy angle but no history, and is trained directly with PPO.

FIG 4教师用特权信息学会控制，学生从历史里学会同样的事The teacher learns control with privileged information; the student learns the same from its history

左：所选策略的均值动作控制关节 2 秒（循环播放），末端圆点的大小表示负载；紫色虚线是指令角，拖动它的末端可以换指令。连杆颜色表示策略：蓝为教师，绿为学生，红为无记忆 PPO。右：三种策略在负载 0、0.5、1 下的测试误差（rad），颜色同左。先按①训练教师（60 次 PPO 迭代、32 个环境），再按②蒸馏学生（4 轮 DAgger，每轮 16 个回合：第 1 轮由教师驾驶，之后由学生驾驶，教师给每个状态打标签，数据累积起来做回归）。读数给出每种策略用掉的环境步数，以及学生每一轮之后的平均误差。 Left: the mean action of the selected policy controls the joint for 2 seconds (looping), and the size of the dot at the tip shows the payload; the purple dashed line is the command, and dragging its end changes it. The link's colour shows the policy: blue the teacher, green the student, red the memoryless PPO policy. Right: the test errors (rad) of the three policies at payloads 0, 0.5 and 1, coloured as on the left. Press ① to train the teacher (60 PPO iterations, 32 environments), then ② to distill the student (4 rounds of DAgger with 16 episodes each: the teacher drives in round 1 and the student afterwards, the teacher labels every state, and the data is aggregated for regression). The readout gives the environment steps each policy used and the student's mean error after every round.

按①：60 次迭代、96000 个环境步之后，教师在三个负载上的平均误差约 0.047 rad（低于 0.07 即完成）。Press ①: after 60 iterations and 96000 environment steps, the teacher's mean error over the three payloads is about 0.047 rad (below 0.07 completes the task).
按②：学生每轮之后的平均误差依次约为 0.39、0.12、0.061、0.047。第 1 轮只看过教师走过的状态（这就是行为克隆），自己开起来误差很大；DAgger 把学生自己会到达的状态补进数据之后，4 轮就追平了教师，只用了 3200 步，不到教师的 1/10（完成条件：误差不超过教师 + 0.02，步数少于教师的 1/10）。Press ②: the student's mean error after each round is about 0.39, 0.12, 0.061 and 0.047. After round 1 it has only seen the states the teacher visits (that is behaviour cloning), and driving on its own it errs badly; once DAgger adds the states the student itself reaches, it matches the teacher in 4 rounds using 3200 steps, less than a tenth of the teacher's (completes when its error is within the teacher's + 0.02 and its steps are under a tenth of the teacher's).
先完成②，再按“对照：无记忆 PPO”：同样 60 次迭代、96000 步，平均误差约 0.23 rad，是学生的 3 倍以上，在负载 0 时尤其差。只看一个带噪声的角度，它既估不出速度，也估不出负载。After ②, press "Baseline: memoryless PPO": with the same 60 iterations and 96000 steps its mean error is about 0.23 rad, more than 3 times the student's, and worst at payload 0. Seeing only a noisy angle, it can estimate neither the speed nor the payload.

同样的历史输入也可以直接交给 PPO 训练（不经过教师）：在这个小例子里，带 4 步历史的策略用 PPO 训练 60 次迭代后也能达到和教师相近的误差，critic 换成看特权状态的非对称 critic，最终结果也差不多。差距在观测离状态很远的时候才显现：Pinto 等（2018）的 actor 看的是图像，Lee 等（2020）的学生要从本体感觉历史里推断地形，这时直接对部分观测做 RL 又慢又不稳定，特权 critic 和教师就成了关键。

The same history could also be handed straight to PPO (no teacher): in this small example, a policy with a 4-step history trained with PPO for 60 iterations also reaches an error close to the teacher's, and swapping in an asymmetric critic that sees the privileged state ends up about the same. The difference shows when the observation is far from the state: the actor of Pinto et al. (2018) sees images, and the student of Lee et al. (2020) must infer the terrain from proprioceptive history; then RL directly on partial observations is slow and unstable, and the privileged critic and the teacher become essential.

现实差距从哪来：系统辨识与执行器网络Where the reality gap comes from: system identification and actuator networks

现实差距有三个主要来源。动力学：质量、惯量、摩擦、接触刚度不准，最难的往往是执行器本身（9.4 节）：减速器摩擦、反电动势带来的“转得越快力矩越小”、驱动器内部回路的带宽。延迟：传感器读数、网络通信、推理、电机驱动加起来几十毫秒，仿真里常常是零；它让系统不再是马尔可夫的，策略需要知道最近发出过什么动作。传感：噪声、偏置、量化、丢帧，以及仿真里干净、真机上满是空洞的深度图。

系统辨识是最直接的对策：在真机上采一段数据，调仿真的参数让它复现这段数据（8.3 节讲过刚体参数是线性可辨识的）。执行器常常没有好用的物理模型，Hwangbo 等（2019）的做法是执行器网络：在真机上记录“目标位置误差和速度的最近历史 → 实际输出力矩”，训练一个 MLP 去拟合，再把它放进仿真循环里代替理想电机。下图用一个“真实”驱动器做这件事：它的力矩随转速下降（反电动势）、有约 15 ms 的滞后、比铭牌弱 15%，测量还带 0.15 N·m 的噪声。

The reality gap has three main sources. Dynamics: masses, inertias, friction and contact stiffness are off, and the hardest part is often the actuator itself (Lesson 9.4): gearbox friction, back-EMF making "less torque the faster it spins", the bandwidth of the loops inside the drive. Latency: sensor readout, communication, inference and the motor drive add up to tens of milliseconds, often zero in simulation; latency makes the system non-Markovian, so the policy needs to know which actions it recently sent. Sensing: noise, bias, quantization and dropped frames, and depth images that are clean in simulation and full of holes on the robot.

System identification is the most direct remedy: record data on the robot and tune the simulator's parameters until it reproduces them (Lesson 8.3 showed that rigid-body parameters are linearly identifiable). Actuators often have no usable physical model; Hwangbo et al. (2019) used an actuator network: record on the robot "the recent history of position-target errors and speeds → the torque actually produced", fit an MLP to it, and put it into the simulation loop in place of the ideal motor. The figure does this for a "real" drive whose torque falls with speed (back-EMF), lags by about 15 ms and is 15% weaker than its datasheet, with 0.15 N·m of noise on the measurement.

FIG 5辨识执行器：理想模型、物理参数拟合、执行器网络Identifying the actuator: the ideal model, a physical-parameter fit, an actuator network

数据：关节在真机上跟踪一串随机目标角（每 0.25 s 换一次，叠加一个慢正弦），200 Hz 记录；6 秒用来拟合，另外 3 秒（不同的随机目标）用来测试。上：测试数据前 1.5 秒的测得力矩（琥珀点）与所选模型的预测（紫线）。下：整段 3 秒测试中真机的关节角（琥珀线）和把所选模型放进仿真、给同样目标时仿真出的关节角（紫线）。三种模型：理想 PD $\tau=K_p(q^{\text{tgt}}-q)-K_d\dot q$；物理参数模型，在理想 PD 外乘一个强度系数、再过一个一阶滞后，两个参数可以用滑块调，也可以按“最小二乘拟合”在网格上找训练数据误差最小的一组；执行器网络，一个 6–16–16–1 的 tanh MLP，输入是当前、10 ms 前、20 ms 前的位置误差和速度（Hwangbo 等的输入配置），按“训练”用 Adam 在拟合数据上实时训练。 Data: the joint tracks a sequence of random targets on the robot (a new one every 0.25 s, plus a slow sine), logged at 200 Hz; 6 seconds are used for fitting and another 3 seconds (different random targets) for testing. Top: the measured torque in the first 1.5 s of the test log (amber dots) and the prediction of the selected model (purple line). Bottom: the robot's joint angle over the whole 3 s test (amber line) and the angle simulated with the selected model for the same targets (purple line). The three models: ideal PD $\tau=K_p(q^{\text{tgt}}-q)-K_d\dot q$; the physical model, which multiplies the ideal PD by a strength factor and passes it through a first-order lag, with both parameters on sliders or found by "Least-squares fit" as the grid point with the smallest error on the fitting data; and the actuator network, a 6–16–16–1 tanh MLP whose inputs are the position error and speed now, 10 ms ago and 20 ms ago (the input configuration of Hwangbo et al.), trained live with Adam on the fitting data by pressing "Train".

理想 PD 的测试力矩误差是 3.69 N·m，仿真与真机的关节角差 0.189 rad。选“物理参数”，用滑块或“最小二乘拟合”把测试误差降到 1.3 N·m 以下。拟合给出强度 0.73、滞后 15 ms：误差 1.07 N·m，角度差 0.066 rad。注意强度比真值 0.85 低：把“真”参数 (0.85, 15 ms) 填进去，误差反而是 1.71。The ideal PD has a test torque error of 3.69 N·m and a sim-vs-robot angle gap of 0.189 rad. Choose "physical" and bring the test error below 1.3 N·m with the sliders or "Least-squares fit". The fit gives strength 0.73 and a 15 ms lag: 1.07 N·m and an angle gap of 0.066 rad. Note that the strength is lower than the true 0.85: entering the "true" parameters (0.85, 15 ms) gives a worse error, 1.71.
按“训练”训练执行器网络，直到测试力矩误差低于 1.0 N·m（约 1250 步，不到一秒）。继续训练几秒（5000 到 12000 步），误差在 0.75 到 0.8 N·m 之间，角度差约 0.02 到 0.04 rad，都比物理模型（1.07 N·m、0.066 rad）好。别训练太久：到 15000 步以后，力矩误差几乎不变，角度差却回升到 0.06 到 0.1 rad，不再胜过物理模型。网络只见过 6 秒的数据，仿真一旦走到数据之外，它的误差就会累积；实践中要用留出的数据决定何时停止。Press "Train" to train the actuator network until its test torque error is below 1.0 N·m (about 1250 steps, under a second). Keep training for a few seconds (5000 to 12000 steps) and the error sits between 0.75 and 0.8 N·m with an angle gap of about 0.02 to 0.04 rad, both better than the physical model (1.07 N·m, 0.066 rad). Do not train too long: from about 15000 steps on, the torque error barely changes but the angle gap climbs back to 0.06–0.1 rad, no longer beating the physical model. The network has seen only 6 seconds of data, and once the simulation leaves that data its errors accumulate; in practice a held-out log decides when to stop.

一个问题，几种看法One problem, several views

辨识出来的参数不是“真值”：物理模型缺了转速对力矩的限制，最小二乘就用一个偏小的强度去补偿它；参数吸收了模型没写进去的效应。这不是错误，而是提醒：辨识得到的是“让这个模型最像真机的参数”。 执行器网络是一个学到的动力学模型：24.1 节会把整个系统的动力学都这样学出来。 域随机化是另一条路（23.2 节）：不去把参数辨识准，而是让策略在一整片参数上都能工作，代价是策略可能变得保守。实践中常常组合使用：先辨识出一个中心值，再在它周围随机化。

Identified parameters are not the "true" values: the physical model lacks the speed limit on torque, so least squares compensates with a lower strength; the parameters absorb effects the model leaves out. That is not a mistake but a reminder: identification yields "the parameters that make this model most like the robot". An actuator network is a learned dynamics model: Lesson 24.1 learns the dynamics of a whole system this way. Domain randomization is the other route (Lesson 23.2): instead of identifying the parameters exactly, make the policy work over a whole range of them, at the risk of a more conservative policy. In practice the two are combined: identify a central value, then randomize around it.

5再看数学The math

提示：把鼠标悬停在公式中带底色的符号上，图中对应的元素会被高亮（FIG 1 的仿真地形边框与真机响应曲线、FIG 2 的指令虚线、FIG 3 的容差曲线、FIG 4 的教师与学生柱、FIG 5 的模型预测曲线）。Tip: hover the highlighted symbols in the formulas; the matching figure elements light up (FIG 1's simulator map frame and robot response, FIG 2's dashed commands, FIG 3's tolerance curve, FIG 4's teacher and student bars, FIG 5's model prediction).

sim-to-real 是在一族 MDP 上做优化Sim-to-real is optimization over a family of MDPs

沿用 20.1 节的记号（第 $t$ 步动作之后得到奖励 $r_{t+1}$），把仿真器的物理参数（质量、摩擦、延迟、电机强度……）记为 $\xi$，转移概率写成 $P_\xi(s'\mid s,a)$。同一个策略 $\pi_\theta$ 在不同 $\xi$ 下有不同的回报 $J(\theta;\xi)=\mathbb E_{\tau\sim\pi_\theta,P_\xi}[\sum_{t\ge0}\gamma^tr_{t+1}]$。在仿真里训练，就是在某个 $\xi_{\text{sim}}$ 上求最优；真机的参数是未知的 $\xi^\star$：

In the notation of Lesson 20.1 (the action at step $t$ earns the reward $r_{t+1}$), write the simulator's physical parameters (masses, friction, latency, motor strength, …) as $\xi$ and its transition probabilities as $P_\xi(s'\mid s,a)$. The same policy $\pi_\theta$ has a different return under different $\xi$: $J(\theta;\xi)=\mathbb E_{\tau\sim\pi_\theta,P_\xi}[\sum_{t\ge0}\gamma^tr_{t+1}]$. Training in simulation optimizes at one $\xi_{\text{sim}}$; the robot has unknown parameters $\xi^\star$:

$$ \theta_{\text{sim}}=\arg\max_\theta\,\hl{cem}{J(\theta;\xi_{\text{sim}})},\qquad J(\theta_{\text{sim}};\xi^\star)=J(\theta_{\text{sim}};\xi_{\text{sim}})-\hl{gap}{\Delta(\theta_{\text{sim}})},\qquad \Delta(\theta)=J(\theta;\xi_{\text{sim}})-J(\theta;\xi^\star) $$

（FIG 1 里用的是代价 $J$，越小越好，符号相反，道理相同。）现实差距 $\Delta$ 这一项正是优化器看不见、也最容易被放大的部分：$\theta_{\text{sim}}$ 是为 $\xi_{\text{sim}}$ 量身定做的，它常常落在 $J(\cdot;\xi_{\text{sim}})$ 很陡、或者紧挨着别的 $\xi$ 下的不稳定区的地方。对策分三类：让 $\xi_{\text{sim}}$ 接近 $\xi^\star$（系统辨识），把目标换成一个分布上的期望 $\mathbb E_{\xi\sim p(\xi)}[J(\theta;\xi)]$（域随机化，23.2 节；FIG 1 的绿色路径），或者让策略的输入里含有足以推断 $\xi$ 的历史，使它对每个 $\xi$ 分别接近最优（自适应策略，下文的教师–学生和 23.2 节的 RMA）。

(FIG 1 uses a cost $J$, lower is better: the sign flips, the argument does not.) The reality gap $\Delta$ is exactly what the optimizer cannot see and what it most easily inflates: $\theta_{\text{sim}}$ is tailored to $\xi_{\text{sim}}$ and often lands where $J(\cdot;\xi_{\text{sim}})$ is steep, or right next to the unstable region of some other $\xi$. The remedies come in three kinds: bring $\xi_{\text{sim}}$ close to $\xi^\star$ (system identification), replace the objective by an expectation over a distribution, $\mathbb E_{\xi\sim p(\xi)}[J(\theta;\xi)]$ (domain randomization, Lesson 23.2; FIG 1's green path), or give the policy enough history to infer $\xi$ so that it is near-optimal for each $\xi$ separately (adaptive policies: the teacher–student scheme below and the RMA of Lesson 23.2).

观测、动作与奖励的设计Designing observations, actions and rewards

动作：位置目标是默认姿态 $\mathbf q_0$ 加上缩放后的网络输出，由高频 PD 回路跟踪；FIG 2 的 $\mathbf q_0=0$、缩放 $s=1$：

Actions: the position target is the default posture $\mathbf q_0$ plus the scaled network output, tracked by a fast PD loop; FIG 2 uses $\mathbf q_0=0$ and scale $s=1$:

$$ \hl{pd}{\mathbf q^{\text{tgt}}_t=\mathbf q_0+s\,\mathbf a_t},\qquad \boldsymbol\tau(t')=\operatorname{sat}\!\big(K_p(\mathbf q^{\text{tgt}}_t-\mathbf q(t'))-K_d\,\dot{\mathbf q}(t')\big),\quad t'\in[t,\,t+\Delta t_{\text{policy}}) $$

PD 回路在策略的两个决策之间仍在用最新的 $\mathbf q,\dot{\mathbf q}$ 做反馈，所以它像一个内环：策略只需要给出“去哪里”，“怎样稳定地去”有一部分由内环负责。$K_p$ 不是越大越好：它太大，策略的小抖动就变成大力矩，太小，跟踪误差会依赖负载。注意，这时策略学到的是“目标角”与“实际角”之间的偏移，重力补偿就藏在这个偏移里（FIG 2 的 PD 策略在指令 1 rad 时输出的目标角明显大于 1）。

观测：本体感觉（关节角、角速度、IMU 姿态与角速度）、指令、上一步的动作；延迟 $d$ 步时，系统在增广状态 $\tilde s_t=(s_t,\mathbf a_{t-1},\dots,\mathbf a_{t-d})$ 上才是马尔可夫的，所以把最近的动作放进观测几乎总是对的。奖励：

Between two policy decisions the PD loop keeps feeding back the latest $\mathbf q,\dot{\mathbf q}$, so it acts as an inner loop: the policy only has to say "where to", and part of "how to get there stably" is handled by the inner loop. A larger $K_p$ is not always better: too large and the policy's small jitters become large torques, too small and the tracking error depends on the load. Note that the policy then learns an offset between the "target" angle and the actual one, and gravity compensation hides in that offset (FIG 2's PD policy sends a target clearly beyond 1 rad when commanded to 1 rad).

Observations: proprioception (joint angles and speeds, IMU orientation and angular rate), the command, and the previous action; with a latency of $d$ steps the system is Markovian only in the augmented state $\tilde s_t=(s_t,\mathbf a_{t-1},\dots,\mathbf a_{t-d})$, so putting recent actions into the observation is almost always right. Rewards:

$$ r_{t+1}=w_{\text{trk}}\,\exp\!\big(-\|\mathbf e_{t+1}\|^2/\sigma^2\big)\;-\;w_\tau\|\boldsymbol\tau_t\|^2\;-\;w_{\Delta a}\|\mathbf a_t-\mathbf a_{t-1}\|^2\;-\;\dots,\qquad r^{\text{sparse}}_{t+1}=\mathbb 1\big[\|\mathbf e_{t+1}\|<\hl{tol}{\varepsilon_k}\big] $$

FIG 3 的课程规则是：命中率 $h_k>0.6$ 时 $\varepsilon_{k+1}=\max(0.8\,\varepsilon_k,\ 0.05)$，否则不变。Hwangbo 等的正则项课程是 $r_{t+1}=r^{\text{task}}_{t+1}-k_c\sum_i w_i c_i$，$k_c\leftarrow k_c^{\,k_d}$。两者的共同点：课程改变目标函数，所以中途的回报曲线不能直接相比，要用一个固定的指标（这里是测试误差）来评估。

FIG 3's curriculum rule is: when the hit rate $h_k>0.6$, $\varepsilon_{k+1}=\max(0.8\,\varepsilon_k,\ 0.05)$, otherwise it stays. Hwangbo et al.'s regularizer curriculum is $r_{t+1}=r^{\text{task}}_{t+1}-k_c\sum_i w_i c_i$ with $k_c\leftarrow k_c^{\,k_d}$. What they share: a curriculum changes the objective, so return curves taken at different stages cannot be compared directly; evaluate with a fixed metric (here the test error).

并行环境里的 PPOPPO over parallel environments

20.4 节的截断目标不变，变的是一次迭代的批量：$N$ 个环境各走 $T$ 步，批量 $B=NT$。梯度估计的方差大致按 $1/B$ 下降，所以 $N$ 增大时，每次迭代的方向更准、可以少迭代几次；但当梯度已经足够准，瓶颈变成“每次迭代能把策略改多少”（受截断 $\epsilon$ 和学习率限制），再加环境就没用了，这就是 FIG 2 在 64 到 128 之间饱和的原因。腿式机器人的常见规模是几千个环境、每个环境每次迭代只走几十步：$T$ 短，GAE 用 $V_\phi$ 在截断处自举（20.4 节误区 1），这正好让 $N$ 可以很大。

The clipped objective of Lesson 20.4 is unchanged; what changes is the batch of one iteration: $N$ environments of $T$ steps each, a batch $B=NT$. The variance of the gradient estimate falls roughly like $1/B$, so a larger $N$ gives a more accurate direction per iteration and fewer iterations; but once the gradient is accurate enough, the bottleneck becomes how far one iteration may move the policy (limited by the clip $\epsilon$ and the learning rate), and more environments no longer help: that is why FIG 2 saturates between 64 and 128. Legged robots typically use thousands of environments that each take only a few dozen steps per iteration: $T$ is short and GAE bootstraps with $V_\phi$ at the cut (Lesson 20.4, Pitfall 1), which is exactly what allows $N$ to be large.

# 机器人 RL 的标准训练循环（伪代码，GPU 上所有环境是一个张量批量）
envs = ParallelSim(N=4096, randomize=dr_ranges)          # 23.2 节：每个环境一组随机物理参数
obs  = envs.reset()
for it in range(n_iters):
    for t in range(T):                                    # T 约 24 步
        a, logp = policy.sample(obs)                      # 演员只看真机也有的观测
        v = critic(envs.privileged_state())               # 非对称：评论家可以看特权状态
        obs, r, done, timeout = envs.step(a)              # 内部：PD 回路跑 decimation 个物理步
        buffer.add(obs, a, logp, r, v, done, timeout)
    adv, ret = gae(buffer, gamma=0.99, lam=0.95)          # 超时截断处用 V 自举
    ppo_update(policy, critic, buffer, adv, ret, clip=0.2, epochs=5)
    curriculum.update(envs.episode_stats())               # 地形等级 / 容差 / 惩罚系数

# the standard robot-RL training loop (pseudo-code; on a GPU all environments are one tensor batch)
envs = ParallelSim(N=4096, randomize=dr_ranges)          # Lesson 23.2: random physics per environment
obs  = envs.reset()
for it in range(n_iters):
    for t in range(T):                                    # T is about 24 steps
        a, logp = policy.sample(obs)                      # the actor sees only what the robot has
        v = critic(envs.privileged_state())               # asymmetric: the critic may see privileged state
        obs, r, done, timeout = envs.step(a)              # inside: the PD loop runs `decimation` physics steps
        buffer.add(obs, a, logp, r, v, done, timeout)
    adv, ret = gae(buffer, gamma=0.99, lam=0.95)          # bootstrap with V at time-outs
    ppo_update(policy, critic, buffer, adv, ret, clip=0.2, epochs=5)
    curriculum.update(envs.episode_stats())               # terrain level / tolerance / penalty factor

（这段伪代码里的数值只是这类代码库常见的量级，不是某篇论文的设置。）

(The numbers in this pseudo-code are only the usual orders of magnitude in such code bases, not the settings of a particular paper.)

非对称 critic 为什么可以，以及它的边界Why an asymmetric critic is allowed, and where it stops

演员的动作只依赖观测历史 $h_t$：$a_t\sim\pi_\theta(\cdot\mid h_t)$。把任何不依赖 $a_t$ 的量当作基线，梯度都无偏（20.3 节）。完整状态 $s_t$ 就是这样一个量：给定 $(h_t,s_t)$，$a_t$ 只通过 $h_t$ 抽取，所以

The actor's action depends only on the observation history $h_t$: $a_t\sim\pi_\theta(\cdot\mid h_t)$. Any quantity that does not depend on $a_t$ can serve as a baseline without biasing the gradient (Lesson 20.3). The full state $s_t$ is such a quantity: given $(h_t,s_t)$, $a_t$ is drawn through $h_t$ alone, so

$$ \mathbb E\big[\nabla_\theta\log\pi_\theta(a_t\mid h_t)\,V_\phi(\hl{priv}{s_t})\big]=\mathbb E_{h_t,s_t}\Big[V_\phi(s_t)\,\underbrace{\mathbb E_{a_t\sim\pi_\theta(\cdot\mid h_t)}\big[\nabla_\theta\log\pi_\theta(a_t\mid h_t)\big]}_{=\,0}\Big]=0 $$

所以把 $V(s_t)$ 当基线从蒙特卡洛回报里减掉，是无偏的，而且因为 $s_t$ 比 $h_t$ 信息多，基线更准，方差更小。但 GAE 和 TD 还把 critic 当自举的目标，这时它应该估计的是“当前这个只看 $h$ 的策略从这里出发的回报”，而这个回报一般同时依赖 $h_t$ 和 $s_t$（同一个状态下，策略因为历史不同会做不同的事）。只用 $s_t$ 的 critic 在部分可观测时一般是有偏的；Baisero 和 Amato（2022）给出了这一点的分析，并提出用 $V(h_t,s_t)$ 作 critic 来消除偏差。实践中特权 critic 通常仍然有效，只要知道它的理论保证在哪里结束。

So subtracting $V(s_t)$ from Monte Carlo returns as a baseline is unbiased, and since $s_t$ carries more information than $h_t$, the baseline is more accurate and the variance lower. But GAE and TD also use the critic as a bootstrapping target, and then it should estimate "the return of this $h$-only policy from here on", which in general depends on both $h_t$ and $s_t$ (in the same state, the policy acts differently depending on its history). A critic of $s_t$ alone is generally biased under partial observability; Baisero and Amato (2022) analysed this and proposed the history–state critic $V(h_t,s_t)$ to remove the bias. In practice privileged critics usually still work; just know where their guarantee ends.

教师–学生蒸馏的目标The objective of teacher–student distillation

教师 $\pi^{\text{T}}$ 用 RL 在 $(o_t,\hl{priv}{x^{\text{priv}}_t})$ 上训练好之后固定。学生的参数 $\psi$ 最小化它和教师在学生自己的状态分布上的差别：

The teacher $\pi^{\text{T}}$, trained with RL on $(o_t,\hl{priv}{x^{\text{priv}}_t})$, is then frozen. The student's parameters $\psi$ minimize its difference from the teacher on the student's own state distribution:

$$ \hl{dagger}{\min_\psi\ \mathbb E_{s_t\sim d^{\pi^{\text{S}}_\psi}}\Big[\big\|\mu^{\text{S}}_\psi(o_{t-K:t},\,a_{t-K:t-1})-\mu^{\text{T}}(o_t,x^{\text{priv}}_t)\big\|^2\Big]} $$

如果期望取在教师的状态分布 $d^{\pi^{\text{T}}}$ 上，这就是行为克隆，误差会像 22.1 节分析的那样随时间累积（FIG 4 第 1 轮之后误差 0.39）；DAgger 交替“用学生采样”和“在累积数据上回归”，把期望移到学生的分布上（第 4 轮 0.047）。这里的回归是监督学习，比 RL 便宜得多：学生用的环境步数不到教师的 1/10。前提是学生的输入里确实含有推断特权信息所需的东西：如果某个特权量无论多长的历史都推断不出来（比如前方还没踩到的地形），学生就学不会教师依赖它的那部分行为，这时要么给学生加上相应的传感器（相机、高度图），要么让教师少依赖它。

Take the expectation over the teacher's distribution $d^{\pi^{\text{T}}}$ instead and this is behaviour cloning, whose error compounds over time as analysed in Lesson 22.1 (FIG 4 after round 1: 0.39); DAgger alternates "sample with the student" and "regress on the aggregated data" and moves the expectation onto the student's distribution (round 4: 0.047). The regression is supervised learning, far cheaper than RL: the student used less than a tenth of the teacher's environment steps. The premise is that the student's input actually contains what is needed to infer the privileged information: if some privileged quantity cannot be inferred from any length of history (the terrain ahead that has not been stepped on yet, say), the student cannot learn the part of the teacher's behaviour that relies on it, and then either the student gets the matching sensor (a camera, a height map) or the teacher must rely on it less.

系统辨识与执行器网络System identification and actuator networks

给一段真机记录 $\{(\mathbf u_i,\tau_i)\}$（$\mathbf u_i$ 是驱动器的输入历史，$\tau_i$ 是测得的力矩），物理参数的辨识与执行器网络都是最小二乘回归，只是模型族不同：

Given a robot log $\{(\mathbf u_i,\tau_i)\}$ ($\mathbf u_i$ the history of the drive's inputs, $\tau_i$ the measured torque), identifying physical parameters and fitting an actuator network are both least-squares regressions, with different model families:

$$ \hat\xi=\arg\min_\xi\sum_i\big(\tau_i-\tau_{\text{model}}(\mathbf u_i;\xi)\big)^2,\qquad \hat\psi=\arg\min_\psi\sum_i\big(\tau_i-\hl{act}{f_\psi(e_t,e_{t-0.01},e_{t-0.02},\dot q_t,\dot q_{t-0.01},\dot q_{t-0.02})}\big)^2 $$

其中 $e=q^{\text{tgt}}-q$。前者参数少、可解释、外推好，但模型结构错了就有偏（FIG 5 的强度偏小）；后者能拟合模型写不出来的效应，但只在采过数据的工况附近可信，所以激励信号要覆盖策略将来会用到的速度和力矩范围。延迟 $d$ 可以单独辨识（比较指令和响应的互相关），然后在仿真里加同样长的缓冲；ANYmal 的执行器网络把驱动器内部的动态都包了进去。

with $e=q^{\text{tgt}}-q$. The former has few parameters, is interpretable and extrapolates well, but is biased when the model structure is wrong (FIG 5's low strength); the latter can fit effects no model writes down, but is trustworthy only near the operating conditions that were recorded, so the excitation must cover the speeds and torques the policy will use later. A latency $d$ can be identified on its own (cross-correlate commands and responses) and then reproduced in simulation with a buffer of the same length; ANYmal's actuator network wraps up the drive's internal dynamics as a whole.

6机器人中的它：里程碑、工具与真机 RLIn robotics: landmarks, tools and real-world RL

下表只列出已核对过原文的事实；其余细节请看原论文。

The table lists only facts checked against the papers; for everything else, read the originals.

工作	机器人	做了什么（本节对应的概念）
Hwangbo 等，Science Robotics 2019	ANYmal 四足	在真机上训练执行器网络、放进仿真；用 TRPO 训练策略，输出关节目标位置；对惩罚项用课程因子 $k_c$。所有训练在一台单 CPU、单 GPU 的电脑上完成，每次不超过 11 小时。真机最高速度 1.5 m/s，比 ANYmal 此前的纪录快 25%；还学会了从复杂姿态摔倒后爬起来。
Lee 等，Science Robotics 2020	ANYmal（两代）	教师策略在仿真里看特权信息（如地形与接触状态）用 RL 训练，学生只看本体感觉的历史并模仿教师；配合自适应的地形课程。零样本部署到泥地、雪地、碎石、植被、流水等自然环境。
Rudin 等，CoRL 2021	ANYmal	在一块工作站 GPU 上用成千上万个并行仿真机器人训练 PPO：平地行走 4 分钟以内，崎岖地形约 20 分钟；“游戏式”的地形课程按表现升降难度；迁移到真机。代码 legged_gym / rsl_rl 成了许多后续工作的起点。
OpenAI 等，“Learning Dexterous In-Hand Manipulation”，2018（后刊于 IJRR 2020）	Shadow 灵巧手	PPO + LSTM 策略，全部在随机化仿真里训练，用非对称 actor-critic（价值网络可看真机拿不到的信息）。384 台各 16 核的机器生成经验、8 块 GPU 优化；全随机化下约 100 年的仿真经验、约 50 小时墙钟。真机上方块连续转到目标朝向的中位数是 13 次（仿真里 50 次）。
OpenAI 等，“Solving Rubik's Cube with a Robot Hand”，2019	Shadow 灵巧手	自动域随机化（ADR，23.2 节）：随机化范围随表现自动扩大。解法序列由 Kociemba 求解器给出，策略负责手内操作。需要 15 次面转动的打乱能成功 60%，最难的 26 次转动的打乱成功 20%。
Zhuang 等，CoRL 2023（Robot Parkour Learning）	Unitree A1 与 Go1	先在“软动力学约束”（允许穿过障碍、逐步收紧）下用 RL 学会攀爬、跳跃、钻爬、侧身挤过等技能，再用 DAgger 把它们蒸馏进一个只用自身深度相机的策略。
Cheng 等，Extreme Parkour，ICRA 2024	Unitree A1	一个直接从前置深度相机图像输出动作的策略：跳上自身高度 2 倍的障碍、跃过自身长度 2 倍的间隙；教师用特权的“scandots”地形信息训练，再蒸馏给看深度图的学生。

Work	Robot	What it did (concepts of this lesson)
Hwangbo et al., Science Robotics 2019	ANYmal quadruped	Trained an actuator network on the robot and put it into the simulator; trained the policy with TRPO, outputting joint position targets; a curriculum factor $k_c$ on the penalty terms. All training ran on one computer with one CPU and one GPU, none longer than 11 hours. Top speed on the robot 1.5 m/s, 25% above ANYmal's previous record; it also learned to get up after falling in complex configurations.
Lee et al., Science Robotics 2020	ANYmal (two generations)	A teacher trained with RL in simulation using privileged information (such as terrain and contact states); a student that sees only proprioceptive history and imitates it; an adaptive terrain curriculum. Deployed zero-shot in natural environments: mud, snow, rubble, vegetation, running water.
Rudin et al., CoRL 2021	ANYmal	PPO over thousands of parallel simulated robots on one workstation GPU: flat-ground walking in under 4 minutes, rough terrain in about 20; a "game-inspired" terrain curriculum that raises and lowers difficulty with performance; transferred to the robot. Its code, legged_gym / rsl_rl, became the starting point of much later work.
OpenAI et al., "Learning Dexterous In-Hand Manipulation", 2018 (later IJRR 2020)	Shadow Dexterous Hand	PPO with an LSTM policy, trained entirely in randomized simulation, with an asymmetric actor-critic (the value network sees information unavailable on the robot). 384 machines with 16 cores each generated experience and 8 GPUs optimized; with all randomizations, about 100 years of simulated experience in about 50 hours of wall-clock time. On the robot, the median number of consecutive successful reorientations of the block was 13 (50 in simulation).
OpenAI et al., "Solving Rubik's Cube with a Robot Hand", 2019	Shadow Dexterous Hand	Automatic domain randomization (ADR, Lesson 23.2): the randomization ranges grow automatically with performance. The solution sequence comes from the Kociemba solver; the policy does the in-hand manipulation. Scrambles needing 15 face rotations succeed 60% of the time, the hardest ones with 26 rotations 20%.
Zhuang et al., CoRL 2023 (Robot Parkour Learning)	Unitree A1 and Go1	Skills (climbing, leaping, crawling, squeezing through slits) learned with RL under "soft dynamics constraints" (obstacles may be penetrated, then the constraint is tightened), then distilled with DAgger into one policy that uses the robot's own depth camera.
Cheng et al., Extreme Parkour, ICRA 2024	Unitree A1	One policy that acts directly on images from a front-facing depth camera: high jumps onto obstacles twice the robot's height, long jumps across gaps twice its length; a teacher trained with privileged "scandots" terrain information, distilled into a depth-image student.

仿真器Simulators

Isaac Gym（Makoviychuk 等，2021）：NVIDIA 的 GPU 物理仿真，物理状态和策略网络都在 GPU 上，通过 PyTorch 张量直接交换数据、不经 CPU，论文报告比“CPU 仿真 + GPU 网络”的常规做法快 2 到 3 个数量级。它已不再维护，继任者是 Isaac Lab（前身 Orbit，Mittal 等，2023），建在 Isaac Sim 之上，带相机、传感器和大量现成的机器人任务。MuJoCo（Todorov 等，2012）是 CPU 上最常用的接触仿真之一；MJX 是它用 JAX 重写的版本（随 MuJoCo 3 发布），可以在 GPU/TPU 上批量运行；MuJoCo Playground（Zakka 等，2025）在 MJX 上提供了四足、人形、灵巧手和机械臂的训练环境，论文称可在单块 GPU 上几分钟内训练出策略，并零样本迁移到真机。Google 的 Brax（Freeman 等，2021）是另一个 JAX 上的可并行仿真器。选哪个，更多取决于你的机器人模型、传感器需求和团队的框架（PyTorch 还是 JAX），而不是谁“更准”。

Isaac Gym (Makoviychuk et al., 2021): NVIDIA's GPU physics simulation, with physics state and policy networks both on the GPU, exchanging data directly as PyTorch tensors without passing through the CPU; the paper reports 2–3 orders of magnitude speed-up over the usual "CPU simulator + GPU network" setup. It is no longer maintained; its successor is Isaac Lab (formerly Orbit, Mittal et al., 2023), built on Isaac Sim, with cameras, sensors and many ready-made robot tasks. MuJoCo (Todorov et al., 2012) is one of the most widely used contact simulators on CPUs; MJX is its reimplementation in JAX (released with MuJoCo 3) that runs batched on GPUs and TPUs; MuJoCo Playground (Zakka et al., 2025) provides training environments on MJX for quadrupeds, humanoids, dexterous hands and arms, and reports training policies in minutes on a single GPU with zero-shot transfer to hardware. Google's Brax (Freeman et al., 2021) is another parallel simulator in JAX. Which one to pick depends more on your robot model, sensor needs and your team's framework (PyTorch or JAX) than on which is "more accurate".

真机上直接学Learning directly on the robot

没有仿真，也能做 RL，只是要换一套算法：真机样本贵，要用 off-policy、可以重复用数据的方法（20.4 节的 SAC）。QT-Opt（Kalashnikov 等，2018）用 7 台机械臂采集了约 58 万次真实抓取，学到的视觉抓取策略对没见过的物体成功率 96%；Haarnoja 等（2019）让 Minitaur 四足在真机上大约两小时学会稳定步态；Smith、Kostrikov 和 Levine（2022）让 Unitree A1 在约 20 分钟内学会行走。难点不在算法，而在工程：自动复位、安全的探索、不会被摔坏的硬件、可靠的奖励测量。今天常见的组合是：仿真里预训练，真机上少量微调或用离线数据（20.5 节）继续改进。

RL without a simulator is possible, but with different algorithms: real samples are expensive, so off-policy methods that reuse data (the SAC of Lesson 20.4) are the choice. QT-Opt (Kalashnikov et al., 2018) collected about 580,000 real grasps with 7 robot arms and learned a vision-based grasping policy with 96% success on unseen objects; Haarnoja et al. (2019) had a Minitaur quadruped learn a stable gait on the hardware in about two hours; Smith, Kostrikov and Levine (2022) had a Unitree A1 learn to walk in about 20 minutes. The hard part is not the algorithm but the engineering: automatic resets, safe exploration, hardware that survives falls, reliable reward measurement. A common combination today is pre-training in simulation and then a little fine-tuning on the robot, or further improvement from offline data (Lesson 20.5).

7常见误区Common pitfalls

误区 1：把仿真里的回报当成真机的表现Pitfall 1: taking the simulated return for real performance

仿真回报只说明策略在 $\xi_{\text{sim}}$ 上好。FIG 1 里仿真最优的增益在 60 ms 延迟下代价翻了 3 倍。部署前至少在一个“留出”的仿真上测：参数偏一点、加延迟、加噪声，看性能掉多少；掉得厉害的策略多半在钻某个缺陷的空子。还要看策略的动作本身：高频抖动、打满力矩、关节速度贴着上限，这些在仿真里不扣分，在真机上会烧电机。

The simulated return only says the policy is good at $\xi_{\text{sim}}$. In FIG 1 the simulator's optimal gains cost three times as much at 60 ms of latency. Before deploying, test at least on a "held-out" simulator: slightly different parameters, added latency, added noise, and see how much performance drops; a policy that drops a lot is probably exploiting some flaw. Look at the actions themselves too: high-frequency chatter, saturated torques, joint speeds riding the limit cost nothing in simulation and burn out motors on the robot.

误区 2：让策略看到真机上没有的东西Pitfall 2: letting the policy see what the robot does not have

仿真里随手可得的量（机身真实线速度、精确的接触状态、无噪声的关节速度）一旦进了演员的输入，部署时就要拿估计值代替，而估计值的误差分布和训练时完全不同。规则很简单：演员的每个输入都要问“真机上它从哪来、多大噪声、多少延迟”，并在仿真里按这个答案去模拟；特权信息只给 critic 或教师。

Quantities that come for free in simulation (the true base velocity, exact contact states, noise-free joint speeds) must be replaced by estimates at deployment once they are in the actor's input, and the estimates' errors are distributed nothing like the training data. The rule is simple: for every actor input ask "where does it come from on the robot, with how much noise and latency", and simulate exactly that; give privileged information only to the critic or the teacher.

误区 3：忽略延迟和控制频率Pitfall 3: ignoring latency and the control rate

仿真里观测和动作是同一时刻的，真机上中间隔着几十毫秒。延迟让系统不再马尔可夫（第 5 节的增广状态），也把高增益策略推向不稳定（FIG 1）。要在仿真里加上与真机一致的延迟（最好再随机化一点），把最近的动作放进观测，保持训练和部署时的控制频率一致，部署时测一下真实的回路延迟。

In simulation, observation and action happen at the same instant; on the robot, tens of milliseconds separate them. Latency makes the system non-Markovian (the augmented state of Section 5) and pushes high-gain policies to instability (FIG 1). Add the robot's latency to the simulator (and randomize it a little), put recent actions into the observation, keep the control rate the same in training and deployment, and measure the real loop latency when you deploy.

误区 4：奖励项越加越多，却不看每一项在做什么Pitfall 4: piling on reward terms without watching what each one does

十几项奖励的权重互相影响，惩罚太重会落进“站着不动”的局部最优，太轻则学出抖动和怪异步态，而且总回报会掩盖这些。逐项记录每个奖励项的平均值，固定一组与奖励无关的评估指标（跟踪误差、能耗、摔倒率），一次只改一个权重；用课程先学任务、再加约束。

A dozen reward weights interact: penalties that are too heavy drop the policy into the "stand still" local optimum, too light ones yield chatter and odd gaits, and the total return hides all of it. Log the mean of every reward term separately, fix a set of evaluation metrics independent of the reward (tracking error, energy, fall rate), change one weight at a time, and use a curriculum to learn the task before the constraints.

误区 5：学生拿不到教师依赖的信息Pitfall 5: a student that cannot get what the teacher relies on

教师–学生蒸馏只在学生的观测“原则上”能推断出教师的特权信息时才行得通。教师若依赖了学生永远推断不出的东西（摩擦系数在没打滑前几乎不可观测、前方地形对盲走的学生不可观测），学生只能学到一个平均行为，常常是两边都不好的折中。对策：限制教师的特权输入，或给学生加传感器；并且像 FIG 4 那样用 DAgger 而不是只在教师的状态上做行为克隆。

Teacher–student distillation works only if the student's observations can, in principle, reveal the teacher's privileged information. If the teacher relies on something the student can never infer (friction is almost unobservable before a slip, the terrain ahead is unobservable to a blind student), the student can only learn an average behaviour, often a compromise that serves neither case. Remedies: restrict the teacher's privileged inputs, or give the student a sensor; and use DAgger as in FIG 4 rather than behaviour cloning on the teacher's states only.

8小练习Check yourself

1. 在 GPU 并行仿真里训练 PPO，把并行环境从 64 增加到 4096，最主要的好处是什么？1. Training PPO in a GPU-parallel simulator, what is the main benefit of going from 64 to 4096 parallel environments?

选项 C。FIG 2 里从 4 到 64 个环境，样本多了 10 倍以上而迭代只少了约三成：样本效率反而下降（排除 A）。并行仿真的意义在于样本几乎免费、墙钟时间才贵。GAE 照用（排除 B）；鲁棒性来自随机化和接口设计，不来自环境数（排除 D）。Option C. In FIG 2, going from 4 to 64 environments costs more than 10 times the samples and saves only about a third of the iterations: sample efficiency actually drops (not A). Parallel simulation pays off because samples are almost free and wall-clock time is expensive. GAE is still used (not B); robustness comes from randomization and interface design, not from the environment count (not D).

2. 为什么腿式机器人的学习策略通常输出关节目标位置（由 PD 回路跟踪），而不是直接输出力矩？2. Why do learned legged-robot policies usually output joint position targets (tracked by a PD loop) rather than torques?

选项 B。FIG 2 里力矩接口在仿真里反而学得快（排除 A），但电机强度降到 0.6 时，它的真机误差是 PD 策略的 2 倍以上，PD 策略仍在 0.05 rad 左右。策略本身仍是非线性网络（排除 C）；很多驱动器可以接受力矩指令（排除 D），PD 目标是一个设计选择。Option B. In FIG 2 the torque interface actually learns faster in simulation (not A), but at motor strength 0.6 its robot error is more than twice the PD policy's, which stays around 0.05 rad. The policy is still a nonlinear network (not C); many drives accept torque commands (not D), and PD targets are a design choice.

3. 非对称 actor-critic 里 critic 看完整状态 $s_t$，actor 只看观测历史 $h_t$。下面哪句话是对的？3. In an asymmetric actor-critic, the critic sees the full state $s_t$ and the actor only the observation history $h_t$. Which statement is right?

选项 D。critic 只在训练时用，部署只需要 actor（排除 A）。基线只要不依赖 $a_t$ 就无偏，第 5 节的推导说明 $s_t$ 满足这一点（排除 B）。但自举时 critic 应当估计依赖历史的策略的回报，这一般同时依赖 $h_t$ 和 $s_t$（Baisero 与 Amato，2022），所以两者并不等价（排除 C）。Option D. The critic is used only in training; deployment needs only the actor (not A). A baseline is unbiased as long as it does not depend on $a_t$, and the derivation in Section 5 shows that $s_t$ qualifies (not B). But when bootstrapping, the critic should estimate the return of a history-dependent policy, which in general depends on both $h_t$ and $s_t$ (Baisero and Amato, 2022), so the two are not equivalent (not C).

4. FIG 4 里学生第 1 轮之后误差约 0.39 rad，第 4 轮之后约 0.047 rad。第 1 轮差的主要原因是？4. In FIG 4 the student's error is about 0.39 rad after round 1 and about 0.047 rad after round 4. What mainly makes round 1 poor?

选项 A。这正是 22.1 节的协变量偏移：网络和历史长度在各轮都一样（排除 B、D），教师在蒸馏开始前已经训练完并固定（排除 C）。DAgger 从第 2 轮起让学生驾驶、教师在学生到达的状态上打标签，把训练分布移到学生自己的分布上。Option A. This is the covariate shift of Lesson 22.1: the network and history length are the same in every round (not B, D), and the teacher is trained and frozen before distillation starts (not C). From round 2 on, DAgger lets the student drive and the teacher label the states it reaches, moving the training distribution onto the student's own.

5. FIG 5 里，最小二乘辨识出的电机强度是 0.73，而真实驱动器的强度是 0.85。最好的解释是？5. In FIG 5, least squares identifies a motor strength of 0.73, while the real drive's strength is 0.85. What is the best explanation?

选项 C。把“真”参数 (0.85, 15 ms) 填进物理模型，测试误差是 1.71 N·m，比拟合值的 1.07 差得多：在这个模型结构下，0.73 才是最好的。数据量与网格步长（0.01）都不是问题（排除 A、B）；测量噪声是零均值的、加在被预测的力矩上，不会系统地把强度往下拉（排除 D）。执行器网络没有这个结构限制，所以误差更低。Option C. Entering the "true" parameters (0.85, 15 ms) into the physical model gives a test error of 1.71 N·m, much worse than the fitted 1.07: within this model structure, 0.73 is the best value. Neither the amount of data nor the grid step (0.01) is the issue (not A, B); the measurement noise is zero-mean and sits on the predicted torque, so it does not pull the strength down systematically (not D). The actuator network has no such structural limit, so its error is lower.

9延伸阅读Further reading

J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun & M. Hutter, Learning Agile and Dynamic Motor Skills for Legged Robots, Science Robotics 4(26), 2019 — 执行器网络、惩罚课程、ANYmal 的 sim-to-realactuator networks, a penalty curriculum, sim-to-real on ANYmal
J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun & M. Hutter, Learning Quadrupedal Locomotion over Challenging Terrain, Science Robotics 5(47), 2020 — 特权教师与本体感觉学生a privileged teacher and a proprioceptive student
N. Rudin, D. Hoeller, P. Reist & M. Hutter, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning, CoRL 2021 — GPU 上数千个并行环境与游戏式课程thousands of parallel environments on a GPU and a game-inspired curriculum
V. Makoviychuk et al., Isaac Gym: High Performance GPU-Based Physics Simulation for Robot Learning, NeurIPS 2021 Datasets and Benchmarks — 端到端在 GPU 上的仿真与训练simulation and training end to end on the GPU
M. Andrychowicz et al. (OpenAI), Learning Dexterous In-Hand Manipulation, IJRR 39(1), 2020 (arXiv:1808.00177) — Dactyl：随机化仿真、非对称 actor-critic、LSTM 策略Dactyl: randomized simulation, asymmetric actor-critic, an LSTM policy
I. Akkaya et al. (OpenAI), Solving Rubik's Cube with a Robot Hand, arXiv:1910.07113, 2019 — 自动域随机化（23.2 节）automatic domain randomization (Lesson 23.2)
L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba & P. Abbeel, Asymmetric Actor Critic for Image-Based Robot Learning, RSS 2018 — critic 看状态、actor 看图像the critic sees the state, the actor sees images
A. Baisero & C. Amato, Unbiased Asymmetric Reinforcement Learning under Partial Observability, AAMAS 2022 — 只看状态的 critic 何时有偏，以及历史–状态 criticwhen a state-only critic is biased, and the history–state critic
D. Chen, B. Zhou, V. Koltun & P. Krähenbühl, Learning by Cheating, CoRL 2019 — 自动驾驶中的特权教师与视觉学生a privileged teacher and a vision-based student for driving
J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez & V. Vanhoucke, Sim-to-Real: Learning Agile Locomotion for Quadruped Robots, RSS 2018 — Minitaur 上的执行器建模、延迟与随机化actuator modelling, latency and randomization on the Minitaur
Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn & H. Zhao, Robot Parkour Learning, CoRL 2023 — 软动力学约束与技能蒸馏soft dynamics constraints and skill distillation
X. Cheng, K. Shi, A. Agarwal & D. Pathak, Extreme Parkour with Legged Robots, ICRA 2024 (arXiv:2309.14341) — 从深度图像直接输出动作的跑酷策略a parkour policy acting directly on depth images
D. Kalashnikov et al., QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation, CoRL 2018 — 大规模真机 RL 抓取large-scale real-world RL for grasping
T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker & S. Levine, Learning to Walk via Deep Reinforcement Learning, RSS 2019 — 在真机上用 SAC 学步态learning a gait on the hardware with SAC
L. Smith, I. Kostrikov & S. Levine, A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning, arXiv:2208.07860, 2022
J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor & S. Levine, How to Train Your Robot with Deep Reinforcement Learning: Lessons We Have Learned, IJRR 40(4–5), 2021 — 真机 RL 的工程经验engineering lessons from real-robot RL
K. Zakka et al., MuJoCo Playground, arXiv:2502.08844, 2025 — 基于 MJX 的 GPU 训练环境与 sim-to-realGPU training environments on MJX and sim-to-real