Part VI · 第 23 章机器人强化学习 · 23.2Part VI · Ch. 23 Robot Reinforcement Learning · 23.2

域随机化 (Domain Randomization)Domain Randomization

仿真器永远不等于真机：负载的质量、电机的延迟、相机看到的颜色和光照，总有几个没测准。域随机化的办法很朴素：不去猜唯一正确的参数，而是在训练时把它们随机地变来变去，让策略在一整片参数分布上都表现得好，真实世界就只是其中一个样本。本节在浏览器里对同一个机械臂关节训练控制器：先看在“标称仿真”上调到最好的控制器为什么一上真机就振荡，再在一个参数分布上训练，并在没见过的参数上测试；然后看随机化范围该多宽（鲁棒性与性能的取舍）、OpenAI 的自动域随机化（ADR）怎样让范围自己长大、RMA 怎样用一段历史隐式地辨识参数从而两头兼得，最后看视觉上的随机化。 A simulator never equals the real robot: the payload mass, the motor latency, the colours and lighting the camera sees, something is always mismeasured. Domain randomization takes a plain approach: instead of guessing the one correct value, keep changing these parameters at random during training, so that the policy does well over a whole distribution of them and the real world is just one more sample. In this lesson you train controllers for one robot arm joint in your browser: first see why the controller tuned best on the "nominal simulator" oscillates on the real robot, then train over a parameter distribution and test on parameters it never saw. Then: how wide the randomization should be (robustness versus performance), how OpenAI's automatic domain randomization (ADR) lets the range grow by itself, how RMA gets the best of both by identifying the parameters implicitly from a short history, and finally randomization of what the camera sees.

约 40 分钟~40 min 前置：23.1 机器人强化学习与 sim-to-real · 21.3 不确定性估计 · 6.11 CEMPrereqs: 23.1 Robot RL & sim-to-real · 21.3 Uncertainty estimation · 6.11 CEM 6 个交互6 interactives

1为什么机器人需要它Why robots need this

23.1 节讲了为什么机器人强化学习几乎都在仿真里做：几千个并行环境一晚上就能产生真机几年的经验，摔坏了也不心疼。代价是现实差距（reality gap）：仿真里的质量、摩擦、电机模型、通信延迟、传感器噪声和渲染出的图像，都只是对真机的近似。23.1 节也讲了缩小差距的第一种办法，系统辨识：把真机的参数测准，再让仿真照着改。可是有些参数测不准（摩擦随温度和磨损变化），有些根本不固定（每次抓起的物体不一样重），还有些你压根没想到要建模。

这时，一个只在“标称参数”上训练的策略会做一件优化器最擅长的事：利用仿真器。如果仿真里电机没有延迟，最优策略就会用很大的反馈增益，因为没有延迟时增益越大跟得越紧；到了有 40 毫秒延迟的真机上，同样的增益会让关节振荡。域随机化（domain randomization，DR）换一种思路：既然不知道真值在哪，那就在训练时对每一局随机抽一组参数，要求策略在整个分布上的平均表现最好。这样的策略没法押宝在某一个参数上，只能学会对参数不敏感的行为，真实世界只要落在分布里，就像又抽到了一个训练样本。

它在机器人学习里几乎无处不在：从 Tobin 等（2017）只用随机纹理的渲染图训练出的物体检测器，到 Peng 等（2018）随机化动力学后直接上真机的推物策略，再到 OpenAI 用自动域随机化训练的、能单手转魔方的 Shadow 灵巧手，以及几乎所有在 GPU 并行仿真里训练、再部署到真机的足式机器人。本节回答四个问题：随机化什么、范围多宽、范围能不能自动定、以及策略能不能不只是“扛住”而是“认出”自己身处哪个环境。

Lesson 23.1 explained why robot reinforcement learning is almost always done in simulation: thousands of parallel environments produce years of real-robot experience overnight, and nothing breaks when they fall. The price is the reality gap: the masses, friction, motor models, communication delays, sensor noise and rendered images of the simulator only approximate the robot. Lesson 23.1 also covered the first way to shrink the gap, system identification: measure the robot's parameters well and correct the simulator to match. But some parameters cannot be measured well (friction drifts with temperature and wear), some are not fixed at all (each grasped object weighs something different), and some you never thought of modelling.

A policy trained only at the "nominal" parameters then does what optimizers do best: it exploits the simulator. If the simulated motor has no delay, the best policy uses very large feedback gains, because without delay a higher gain always tracks more tightly; on the real robot with 40 milliseconds of delay the same gains make the joint oscillate. Domain randomization (DR) turns the problem around: since we do not know where the true values are, draw a fresh random set of parameters for every episode during training and ask for the best average performance over the whole distribution. Such a policy cannot bet on any single parameter value; it has to learn behaviour that is insensitive to them, and as long as the real world falls inside the distribution it looks like one more training sample.

It is everywhere in robot learning: from the object detectors Tobin et al. (2017) trained only on renders with random textures, to the pushing policy Peng et al. (2018) moved straight to a real arm after randomizing its dynamics, to OpenAI's Shadow hand that turned a Rubik's cube with automatic domain randomization, to nearly every legged robot trained in GPU-parallel simulation and deployed on hardware. This lesson answers four questions: what to randomize, how wide, whether the range can be set automatically, and whether a policy can do more than "withstand" the variation, namely recognize which environment it is in.

2直觉：一个控制器，两个世界Intuition: one controller, two worlds

整节课都用同一个任务。一个长 0.5 m、自重 1 kg 的机械臂关节（和 20.3、20.4 节一样，角度 $q$ 从竖直下垂开始量），末端挂着质量为 $m$ 的负载，要在 2 秒内从 $q=0.7$ rad（约 40°）抬到并保持在目标 $q^\star=1.3$ rad（约 74°）。电机力矩上限 30 N·m，控制频率 100 Hz，但发出的力矩命令要过 $d$ 毫秒才真正作用到关节上（总线通信、驱动器滤波、电机电流环加起来的延迟）。环境参数就是这两个数：$\boldsymbol\xi=(m,d)$。

策略是一个只有 4 个参数的反馈控制器：$\tau=K_p e-K_d\dot q+K_{ff}\sin q^\star+K_i\!\int e\,dt$，其中 $e=q^\star-q$。前两项是 PD，第三项是按“标称负载”预估的重力补偿，第四项是积分，用来消掉补偿不准留下的稳态误差。它小到能看清每个参数的意义，但域随机化的道理和用神经网络策略完全一样（第 6 节讲真实系统）。“训练”就是用 6.11 节的 CEM 搜这 4 个增益，代价是一局的 $J=$平均角度误差（度）$+\;0.2\times$平均力矩变化量（N·m），后一项就是 23.1 节说的“动作变化率惩罚”，越小越好。下面的“专才”控制器是在标称仿真（$m=1$ kg、没有延迟）上训练出来的，它在那里的代价是 1.98。23.1 节的 FIG 1 已经在一个水平关节上演示过优化器怎样利用“没有延迟”的仿真；这里在本节一直使用的关节上再看一次，并多加一个参数（负载），好比较两种参数错误的后果。拖动真机的负载和延迟，看看它在另一个世界里会怎样。

The whole lesson uses one task. A robot arm joint 0.5 m long with 1 kg of its own mass (angle $q$ measured from hanging straight down, as in Lessons 20.3 and 20.4) carries a payload of mass $m$ at its tip and must swing from $q=0.7$ rad (about 40°) up to the target $q^\star=1.3$ rad (about 74°) and hold it, within 2 seconds. The motor torque is limited to 30 N·m and the controller runs at 100 Hz, but each torque command reaches the joint only $d$ milliseconds later (bus communication, driver filtering and the motor's current loop add up to a latency). The environment parameters are these two numbers: $\boldsymbol\xi=(m,d)$.

The policy is a feedback controller with just 4 parameters: $\tau=K_p e-K_d\dot q+K_{ff}\sin q^\star+K_i\!\int e\,dt$ with $e=q^\star-q$. The first two terms are PD, the third is a gravity compensation guessed for a "nominal" payload, and the fourth is an integral that removes the steady-state error left when that guess is off. It is small enough that you can see what every parameter does, yet the logic of domain randomization is exactly the same as with a neural-network policy (real systems in Section 6). "Training" means searching these 4 gains with the CEM of Lesson 6.11; the cost of an episode is $J=$ mean angle error (degrees) $+\;0.2\times$ mean change of torque per step (N·m), the second term being the "action-rate penalty" of Lesson 23.1; lower is better. The "specialist" below was trained on the nominal simulator ($m=1$ kg, no latency), where its cost is 1.98. FIG 1 of Lesson 23.1 already showed an optimizer exploiting a delay-free simulator on a horizontal joint; here we look again on the joint used throughout this lesson, with a second parameter (the payload) added so that the consequences of the two kinds of parameter error can be compared. Drag the real robot's payload and latency and see how it fares in the other world.

FIG 1现实差距：同一个控制器，仿真与“真机”The reality gap: one controller, simulator and "real robot"

左：两个关节实时重放同一个控制器的 2 秒（结束后停半秒再重来）。左边是标称仿真，右边是“真机”；琥珀色是机械臂，末端圆的大小表示负载质量，绿色虚线是目标角度。右上：角度随时间变化，青色为仿真，琥珀色为真机，绿色虚线为目标。右下：真机上的力矩，琥珀色是控制器发出的命令，紫色虚线是 $d$ 毫秒之后真正作用到关节上的力矩（没有延迟时两者重合）。 Left: two joints replay the same controller's 2 seconds in real time (then pause half a second and repeat). On the left the nominal simulator, on the right the "real robot"; the arm is amber, the size of the tip disc shows the payload mass, and the green dashed line is the target angle. Top right: angle over time, cyan for the simulator, amber for the real robot, green dashed for the target. Bottom right: torque on the real robot; amber is the command the controller sends, the purple dashed line is the torque that actually reaches the joint $d$ milliseconds later (the two coincide without latency).

试一试Try this

保持真机负载为 1 kg（与仿真相同），只把延迟往上拉，直到真机的代价超过仿真的 3 倍。这发生在多少毫秒？关节在做什么？Keep the real payload at 1 kg (the same as the simulator) and raise only the latency until the real cost exceeds 3 times the simulator's. At how many milliseconds does that happen, and what is the joint doing?
把延迟设回 0，负载加到 3.5 kg 以上。代价也变差了，但曲线的样子和上一条完全不同：差在哪里？Set the latency back to 0 and raise the payload to 3.5 kg or more. The cost also gets worse, but the curve looks completely different from the previous task: how?

核心洞见Key insight

只看延迟：负载不变，延迟到 30 ms 时代价是 3.53，35 ms 时跳到 6.82，40 ms 时 10.19，50 ms 时 20.75：专才用的大增益（$K_p\approx122$）在延迟下失去相位裕度，关节开始持续振荡。只看负载：没有延迟、负载 4 kg 时代价是 6.96，但曲线只是抬得慢一点、略微超调，然后停在目标下方约 3° 处，由积分项慢慢往回拉，没有振荡。两种参数错误的后果形态不同：质量错了是性能慢慢变差，延迟错了是稳定性突然丧失。专才在标称点上无可挑剔，因为它恰恰利用了“没有延迟”这个仿真里的假象。

Latency alone: with the payload unchanged, the cost is 3.53 at 30 ms, jumps to 6.82 at 35 ms, 10.19 at 40 ms and 20.75 at 50 ms: the specialist's large gains ($K_p\approx122$) lose their phase margin under delay and the joint starts to oscillate persistently. Mass alone: with no latency and a 4 kg payload the cost is 6.96, but the curve merely rises more slowly, overshoots a little, and settles about 3° short of the target, where the integral term pulls it in only slowly; it does not oscillate. The two kinds of parameter error fail differently: a wrong mass degrades performance gradually, a wrong latency destroys stability abruptly. The specialist is flawless at the nominal point precisely because it exploits a fiction of the simulator, the absence of delay.

3动手：在一个分布上训练，在没见过的参数上测试Play: train over a distribution, test on parameters never seen

现在把训练改成域随机化。在参数平面（横轴负载 $m\in[0,4]$ kg，纵轴延迟 $d\in[0,80]$ ms）上画一个矩形，这就是训练分布 $\hl{box}{P_\phi}$：在矩形里均匀抽样。CEM 的每一次迭代从矩形里抽 6 组参数，让 24 个候选控制器都在这 6 组参数上各跑一局（大家用同一组样本，比较才公平），取平均代价最低的 5 个更新分布；共 40 次迭代，即 5760 局。训练只看矩形里的参数，从不看“真机”。

训练完后，在 $17\times17=289$ 组没参与训练的参数上测试：每格是控制器在这组参数下一局的代价。真机是图上那颗可拖动的紫色星 $\hl{real}{\boldsymbol\xi^\star}$，默认在 $m=2.5$ kg、$d=50$ ms，也就是“负载比标称重，通信比想象中慢”。

Now switch the training to domain randomization. Draw a rectangle on the parameter plane (horizontal: payload $m\in[0,4]$ kg; vertical: latency $d\in[0,80]$ ms): this is the training distribution $\hl{box}{P_\phi}$, uniform over the rectangle. Each CEM iteration draws 6 parameter pairs from the rectangle, runs all 24 candidate controllers for one episode on each of the 6 (all on the same draws, so the comparison is fair), and refits the search distribution to the 5 with the lowest average cost; 40 iterations, 5760 episodes in all. Training only ever sees parameters inside the rectangle, never the "real robot".

After training, the controller is tested on $17\times17=289$ parameter pairs that took no part in training: each cell is the cost of one episode at that pair. The real robot is the draggable purple star $\hl{real}{\boldsymbol\xi^\star}$, by default at $m=2.5$ kg and $d=50$ ms: "the payload is heavier than nominal and the bus slower than you thought".

FIG 2域随机化：拖出训练范围，训练，再在整个平面上测试Domain randomization: drag the training range, train, test on the whole plane

左：测试图。每格绿色表示成功（$J<6$，越绿代价越低），红色表示失败（越红代价越高）；琥珀色矩形是训练范围，拖它的两个角（空心圆）改变范围；紫色星是真机，可拖动；青色小圆是标称仿真 $(1\text{ kg},0\text{ ms})$。改动范围会重置训练。右上：当前控制器在真机上（琥珀色）和标称仿真上（青色）的角度曲线，绿色虚线为目标。右下：CEM 每次迭代时均值控制器在该次 6 组样本上的平均代价。 Left: the test map. A green cell is a success ($J<6$; greener = lower cost), a red cell a failure (redder = higher cost); the amber rectangle is the training range, resized by dragging its two corners (hollow circles); the purple star is the real robot and can be dragged; the small cyan circle is the nominal simulator $(1\text{ kg},0\text{ ms})$. Changing the range resets training. Top right: the current controller's angle on the real robot (amber) and on the nominal simulator (cyan), target green dashed. Bottom right: the mean controller's average cost on each CEM iteration's 6 draws.

选预设“不随机化”（范围缩成标称点），训练到 40 次迭代。真机星处的代价超过 10，测试图里成功的格子不到三分之一。Choose the preset "no randomization" (the range shrinks to the nominal point) and train for 40 iterations. The cost at the real-robot star exceeds 10, and fewer than a third of the test cells succeed.
自己拖出一个包含紫色星的矩形，训练到 40 次迭代，让真机代价降到 6 以下。Drag a rectangle that contains the purple star, train for 40 iterations and get the real-robot cost below 6.
选预设“全部范围”并训练到 40 次迭代：真机成功了，但标称仿真上的代价是多少？和“不随机化”时相比是几倍？Choose the preset "full range" and train for 40 iterations: the real robot succeeds, but what is the cost on the nominal simulator now? How many times the "no randomization" value is it?

读数说明了什么What the readouts say

默认种子下：“不随机化”得到 $K_p\approx122$，标称点代价 1.98，真机 15.70，成功格子 30%；“全部范围”（$[0,4]$ kg $\times[0,80]$ ms）得到 $K_p\approx37$、$K_{ff}\approx6.9$，真机 4.95，成功格子升到 55%，但标称点代价变成 5.24，是原来的 2.6 倍。这就是域随机化的基本交易：用平均意义上的“哪儿都还行”，换掉某一处的“最好”。还要注意左上角（轻负载、长延迟）无论怎么训练都是红的：负载轻时惯量小，同样的增益更容易被延迟弄得不稳定，而一个固定增益的控制器无法同时照顾到它和右下角的重负载。第 4 节会回到这一点。

With the default seed: "no randomization" gives $K_p\approx122$, cost 1.98 at the nominal point, 15.70 on the real robot, 30% of cells succeeding; "full range" ($[0,4]$ kg $\times[0,80]$ ms) gives $K_p\approx37$ and $K_{ff}\approx6.9$, 4.95 on the real robot and 55% of cells succeeding, but the nominal cost rises to 5.24, 2.6 times the specialist's. This is the basic bargain of domain randomization: trade "the best" at one point for "good enough everywhere" on average. Notice also that the top-left corner (light payload, long latency) stays red however you train: a light payload means low inertia, so the same gains are destabilized by delay more easily, and a controller with fixed gains cannot serve that corner and the heavy payloads at the bottom right at once. Section 4 comes back to this.

4再看四个方面：多宽、自动定范围、自适应、视觉Four more views: how wide, automatic ranges, adaptation, vision

范围多宽？鲁棒性与性能的取舍How wide? The robustness–performance trade-off

FIG 2 里你只试了几个矩形。下面系统地扫一遍：让训练范围从标称点按比例长大，宽度 $w\in[0,1]$ 时范围是 $m\in[1-w,\,1+3w]$ kg、$d\in[0,\,80w]$ ms（$w=1$ 就是全部范围）。对 7 个宽度各训练一次（同一个种子、40 次迭代），记下三个数：标称点的代价、真机（$2.5$ kg、50 ms）的代价，以及 289 个测试格子里成功的比例。

In FIG 2 you tried a few rectangles. Below the range is swept systematically: it grows in proportion from the nominal point, and at width $w\in[0,1]$ it is $m\in[1-w,\,1+3w]$ kg and $d\in[0,\,80w]$ ms ($w=1$ is the full range). For each of 7 widths one controller is trained (same seed, 40 iterations) and three numbers are recorded: the cost at the nominal point, the cost on the real robot ($2.5$ kg, 50 ms), and the fraction of the 289 test cells that succeed.

FIG 3随机化宽度扫描：标称性能、真机性能与覆盖率Sweeping the randomization width: nominal performance, real-robot performance, coverage

横轴是宽度 $w$。青色是标称点的代价 $\hl{nom}{J(\theta_w;\boldsymbol\xi_0)}$，紫色是真机的代价 $J(\theta_w;\boldsymbol\xi^\star)$（左轴，代价，超过 20 的画在顶边），绿色是测试格子的成功比例（右轴）。灰色虚线是成功阈值 $J=6$。按“运行扫描”后 7 个控制器依次训练；点击图中任意位置选中最近的宽度，读数给出它的训练范围和三个数。 The horizontal axis is the width $w$. Cyan is the cost at the nominal point $\hl{nom}{J(\theta_w;\boldsymbol\xi_0)}$, purple the cost on the real robot $J(\theta_w;\boldsymbol\xi^\star)$ (left axis, cost; values above 20 are drawn at the top edge), green the fraction of successful test cells (right axis). The grey dashed line is the success threshold $J=6$. "Run sweep" trains the 7 controllers one after another; click anywhere in the plot to select the nearest width, and the readout gives its training range and the three numbers.

扫描完成后，选中能让真机成功（紫色低于 6）的最小宽度。在那里标称代价付出了多少？When the sweep has finished, select the smallest width at which the real robot succeeds (purple below 6). How much nominal cost has been paid there?

三条曲线讲了一个完整的故事。标称代价基本单调上升：1.98（$w=0$）、2.31（$w=\tfrac12$）、3.09（$w=\tfrac23$）、5.24（$w=1$），范围越宽，控制器越保守。（$w=\tfrac16$ 处的 1.93 比专才的 1.98 还略低一点：CEM 只找到近似最优，专才本身也不是精确的最优解，差这几个百分点在搜索误差之内。）真机代价在两个最窄的范围上都是 15.7，$w=\tfrac13$ 时 11.8，直到范围宽到差不多碰到真机才降下来：$w=\tfrac12$ 时（延迟上限 40 ms）是 6.14，$w=\tfrac23$ 时（上限约 53 ms，真机 50 ms 已经在里面）是 5.09。覆盖率从 30% 涨到 $w=\tfrac56$ 时的 59%，到 $w=1$ 反而回落到 55%：把根本做不好的左上角也塞进训练，平均代价被它主导，控制器为了它进一步降低增益，结果在其它地方都变差了一点。

所以“多宽”的答案是：刚好包住真实参数可能在的地方，再留一点余量，而不是越宽越好。范围应该来自你对真机的认识：测得出的参数（质量、几何）用测量值附近的窄范围，测不准或会变的参数（摩擦、延迟、负载）用宽一点的范围。这正是 21.3 节的认知不确定性：随机化范围描述的是“我们对真机知道多少”，知道得越多，范围就可以越窄，代价就越小。

The three curves tell a complete story. The nominal cost rises almost monotonically: 1.98 ($w=0$), 2.31 ($w=\tfrac12$), 3.09 ($w=\tfrac23$), 5.24 ($w=1$); the wider the range, the more conservative the controller. (The 1.93 at $w=\tfrac16$ is slightly below the specialist's 1.98: CEM finds only approximate optima, the specialist included, and a few percent is within its search error.) The real-robot cost is 15.7 for the two narrowest ranges and 11.8 at $w=\tfrac13$, and falls only once the range almost reaches the robot: 6.14 at $w=\tfrac12$ (latency up to 40 ms) and 5.09 at $w=\tfrac23$ (up to about 53 ms, so the robot's 50 ms is now inside). The coverage climbs from 30% to 59% at $w=\tfrac56$, then falls back to 55% at $w=1$: pushing the hopeless top-left corner into training lets it dominate the average, the controller lowers its gains further for its sake, and gets a little worse everywhere else.

So the answer to "how wide" is: just enough to enclose where the real parameters might be, plus a margin, not as wide as possible. The range should come from what you know about the robot: parameters you can measure (masses, geometry) get a narrow range around the measurement, parameters you cannot measure or that change (friction, latency, payload) a wider one. This is the epistemic uncertainty of Lesson 21.3: the randomization range describes how much we know about the robot, and the more we know, the narrower the range can be and the less performance we give away.

自动域随机化（ADR）：让范围自己长大Automatic domain randomization (ADR): let the range grow by itself

手工调范围在两个参数时还行，可 OpenAI 的魔方手要随机化几十类参数，人根本调不过来。他们在 2019 年提出的自动域随机化（ADR）把范围本身变成训练过程的一部分：每个参数 $i$ 有下界 $\hl{bound}{\phi_i^L}$ 和上界 $\hl{bound}{\phi_i^H}$，一开始两者都等于标定值（范围是一个点）。训练中，ADR 不时做一次边界采样：随机挑一个参数的一条边，把这个参数固定在这条边上，其余参数照常在范围里抽，跑一局，把成绩记进这条边的缓冲区。缓冲区攒够 $M$ 局就求平均：成绩比上阈值 $t_H$ 好，这条边向外推一步 $\Delta$；比下阈值 $t_L$ 差，就向内收一步。于是范围只在策略已经能应付的方向上长大，形成一个自动课程：先在一个环境里学会，再一点点变难。

下图在本节的任务上运行 ADR：三条可动的边是负载下界、负载上界和延迟上界（延迟下界固定为 0，负延迟没有物理意义）。每次迭代先做一次 CEM 更新（在当前范围里抽样），再做 3 次边界采样；$M=4$，$\Delta$ 为 0.1 kg 或 2 ms，成绩用代价衡量，$t_H$ 是“代价 ≤ $t_H$ 就扩”，$t_L=7$ 是“代价 ≥ 7 就收”。

Tuning ranges by hand is fine for two parameters, but OpenAI's Rubik's-cube hand randomized dozens of kinds of parameters, far more than anyone can tune. Their 2019 automatic domain randomization (ADR) makes the ranges part of training. Each parameter $i$ has a lower bound $\hl{bound}{\phi_i^L}$ and an upper bound $\hl{bound}{\phi_i^H}$, both starting at the calibrated value (the range is a point). During training ADR now and then does boundary sampling: pick one bound of one parameter at random, fix that parameter at the bound, draw the others from their ranges as usual, run an episode, and add its performance to that bound's buffer. Once the buffer holds $M$ episodes it is averaged: if performance beats the upper threshold $t_H$ the bound moves outward by a step $\Delta$, if it is worse than the lower threshold $t_L$ it moves inward. The range therefore grows only in directions the policy can already handle, an automatic curriculum: master one environment first, then make it gradually harder.

The figure runs ADR on this lesson's task. The three movable bounds are the payload's lower and upper bounds and the latency's upper bound (the latency's lower bound stays at 0; negative delay is not physical). Each iteration does one CEM update (drawing from the current range) and then 3 boundary samples; $M=4$, $\Delta$ is 0.1 kg or 2 ms, and performance is measured by the cost: "expand if the cost is ≤ $t_H$", "shrink if the cost is ≥ $t_L=7$".

FIG 4ADR：范围从一个点开始自己长大ADR: the range grows by itself, starting from a point

左：参数平面，琥珀色矩形是 ADR 当前的范围 $P_\phi$，它的三条可动边画得更粗；点是最近 60 次边界采样，绿表示代价 ≤ $t_H$，红表示代价 ≥ $t_L$，青色表示两者之间；较大的青色圆点是标称仿真 $(1\text{ kg},0\text{ ms})$，也是范围的起点；空心的紫色星是 FIG 2 的真机（只作参照，ADR 看不见它）。右：ADR 熵（每维平均的对数宽度，宽度以全范围为单位，见第 5 节）随迭代的变化。最多运行 400 次迭代。 Left: the parameter plane; the amber rectangle is ADR's current range $P_\phi$, with its three movable sides drawn thicker; the dots are the last 60 boundary samples, green for cost ≤ $t_H$, red for cost ≥ $t_L$, cyan in between; the larger cyan dot is the nominal simulator $(1\text{ kg},0\text{ ms})$, where the range starts; the hollow purple star is FIG 2's real robot (for reference only; ADR cannot see it). Right: the ADR entropy (the mean log width per dimension, widths in units of the full range; see Section 5) over iterations. It runs for at most 400 iterations.

用 $t_H=4.5$ 运行至少 300 次迭代。负载范围长到多宽？延迟上界停在哪里，为什么不再长？Run at least 300 iterations with $t_H=4.5$. How wide does the payload range grow? Where does the latency bound stop, and why does it not grow further?
把 $t_H$ 改成更严格的 3.5，重置后运行至少 300 次迭代。延迟上界停得更早还是更晚？真机（50 ms）的代价变成了多少？Change $t_H$ to the stricter 3.5, reset and run at least 300 iterations. Does the latency bound stop earlier or later? What is the real robot's (50 ms) cost now?

默认种子下，$t_H=4.5$ 运行 400 次后范围是 $m\in[0.3,3.8]$ kg、$d\in[0,42]$ ms，真机代价 5.73，标称代价 2.66：真机的 50 ms 其实在范围之外，但离边界只有 8 ms，性能缓慢下降，仍然成功；$t_H=3.5$ 时范围只到 $[0.2,2.9]$ kg $\times[0,32]$ ms，真机代价 8.44（失败）；$t_H=5.5$ 时长到 $[0.4,4]$ kg $\times[0,50]$ ms。负载方向几乎长满，延迟方向却停在 30 到 50 ms：再往上，即使范围里的其它参数都取容易的值，这个 4 参数控制器也达不到阈值，边界采样的成绩时好时坏，边就在这里来回抖动。ADR 的范围因此是一张能力地图：它长到策略能力的边缘为止。它不需要知道真机在哪，但也因此不保证覆盖真机：阈值和策略容量决定了它最终停在哪里。

With the default seed, $t_H=4.5$ ends after 400 iterations with $m\in[0.3,3.8]$ kg and $d\in[0,42]$ ms, a real-robot cost of 5.73 and a nominal cost of 2.66: the robot's 50 ms actually lies outside the range, but only 8 ms past the boundary, so performance degrades gently and it still succeeds; with $t_H=3.5$ the range only reaches $[0.2,2.9]$ kg $\times[0,32]$ ms and the real robot costs 8.44 (failure); with $t_H=5.5$ it grows to $[0.4,4]$ kg $\times[0,50]$ ms. The payload direction fills almost completely, but the latency direction stops between 30 and 50 ms: beyond that, even when the other parameter takes easy values, this 4-parameter controller cannot meet the threshold, the boundary results alternate between good and bad, and the bound jitters there. ADR's range is thus a map of capability: it grows up to the edge of what the policy can do. It needs no knowledge of where the real robot is, but for the same reason it does not promise to cover it: the thresholds and the policy's capacity decide where it stops.

自适应策略：从历史里认出自己在哪个世界（RMA）Adaptive policies: recognize the world from history (RMA)

到目前为止的控制器都是无记忆的：它只看当前状态，所以只能找一组对所有参数都“还行”的增益。可是参数在一局之内通常不变，而一局开头的几百毫秒里，关节怎样回应力矩，就已经透露了负载有多重、延迟有多长。如果策略能读最近的一段历史，它就能先“认出”环境，再用这个环境下最合适的动作：这就是隐式系统辨识。

Kumar、Fu、Pathak 与 Malik 的 RMA（Rapid Motor Adaptation，RSS 2021）把这件事拆成两个阶段。阶段 1：在仿真里训练一个能看到“特权”环境参数 $e_t$ 的基础策略 $\pi(x_t,a_{t-1},z_t)$，其中 $z_t=\mu(e_t)$ 是参数的低维编码（他们叫 extrinsics）。阶段 2：真机上没有 $e_t$，于是训练一个适应模块 $\hl{zhat}{\hat z_t}=\phi(x_{t-k:t-1},a_{t-k:t-1})$，只看最近 $k$ 步的状态和动作，用监督回归去预测 $z_t$。部署时 $\phi$ 慢一点地更新 $\hat z_t$，基础策略用它出动作。这和 23.1 节第 4 节的教师–学生蒸馏是近亲：那里学生模仿教师的动作，这里适应模块只回归教师用的隐变量 $z_t$，基础策略原封不动地复用。

下图在本节任务上做同样的两步。阶段 1：基础策略的 4 个增益随 $z=(m/4,\,d/80)$ 线性变化（每个增益 3 个系数，共 12 个），在全部范围上用 CEM 训练，训练时直接告诉它真实的 $z$（就像 RMA 里 $\mu$ 取恒等映射）。阶段 2：适应模块是一个 $90\to32\to32\to2$ 的 tanh 网络，输入最近 30 步（0.3 s）的 $(q-q^\star,\ \dot q/5,\ \tau/30)$，每 10 步（10 Hz）更新一次 $\hat z$；还没攒够 30 步时历史用初始状态补齐，第一次估计之前用范围中心 $\hat z=(0.5,0.5)$。和 RMA 一样，训练数据是同策略采的：用基础策略加上 $\phi$ 自己当前的估计去跑，再用真实的 $z$ 当标签，这样 $\phi$ 见过自己估错时机械臂会怎样走（这和 22.1 节 DAgger 的道理相同）。作为对照，“鲁棒”策略是 FIG 2 里“全部范围”训练出的固定增益控制器，“先知”是直接把真实 $z$ 给基础策略。

Every controller so far has been memoryless: it sees only the current state, so it can only find one set of gains that is "fine" for every parameter. Yet the parameters usually do not change within an episode, and the first few hundred milliseconds of how the joint responds to torque already reveal how heavy the payload is and how long the delay. A policy that reads its recent history can first "recognize" the environment and then act as that environment requires: implicit system identification.

RMA (Rapid Motor Adaptation) by Kumar, Fu, Pathak and Malik (RSS 2021) splits this into two phases. Phase 1: in simulation, train a base policy $\pi(x_t,a_{t-1},z_t)$ that sees the "privileged" environment parameters $e_t$ through a low-dimensional encoding $z_t=\mu(e_t)$ (they call it the extrinsics). Phase 2: the real robot has no $e_t$, so train an adaptation module $\hl{zhat}{\hat z_t}=\phi(x_{t-k:t-1},a_{t-k:t-1})$ that sees only the last $k$ states and actions and regresses $z_t$ by supervised learning. At deployment $\phi$ updates $\hat z_t$ at a slower rate and the base policy acts on it. This is a close relative of the teacher–student distillation in Section 4 of Lesson 23.1: there the student imitates the teacher's actions; here the adaptation module regresses only the teacher's latent $z_t$, and the base policy is reused unchanged.

The figure does the same two steps on this lesson's task. Phase 1: the base policy's 4 gains vary linearly with $z=(m/4,\,d/80)$ (3 coefficients per gain, 12 in all) and are trained by CEM over the full range, being told the true $z$ (as if RMA's $\mu$ were the identity). Phase 2: the adaptation module is a $90\to32\to32\to2$ tanh network whose input is the last 30 steps (0.3 s) of $(q-q^\star,\ \dot q/5,\ \tau/30)$, and it updates $\hat z$ every 10 steps (10 Hz); before 30 steps exist the history is padded with the initial state, and before the first estimate $\hat z=(0.5,0.5)$, the centre of the range. As in RMA, the training data is collected on-policy: run the base policy with $\phi$'s own current estimates and label with the true $z$, so that $\phi$ has seen how the arm moves when it guesses wrong (the same reasoning as DAgger in Lesson 22.1). For comparison, "robust" is the fixed-gain controller trained on the "full range" in FIG 2, and the "oracle" hands the true $z$ to the base policy.

FIG 5RMA：特权基础策略 + 从历史估计参数的适应模块RMA: a privileged base policy + an adaptation module that estimates the parameters from history

先按“阶段 1”，完成后再按“阶段 2”（两者合计在浏览器里约十几秒）。左：适应模块在这一局里的估计 $\hat z$，换算成负载（上）和延迟（下），琥珀色是估计，绿色虚线是真值。右：三个策略的角度曲线，紫色为鲁棒（固定增益），琥珀色为 RMA，青色虚线为先知，绿色虚线为目标。滑块设定这一局的真实参数；“中途加载”在 $t=1$ s 时把负载加 2 kg（上限 4 kg）。读数最后一行是在 FIG 2 的 289 个测试格子上三者的成功比例与平均代价。 Press "Phase 1", and when it is done "Phase 2" (together about a dozen seconds in the browser). Left: the adaptation module's estimate $\hat z$ during this episode, converted to payload (top) and latency (bottom); amber is the estimate, green dashed the truth. Right: the angle of the three policies, purple robust (fixed gains), amber RMA, cyan dashed the oracle, green dashed the target. The sliders set this episode's true parameters; "pick up mid-episode" adds 2 kg to the payload at $t=1$ s (capped at 4 kg). The last line of the readout gives the three policies' success fraction and mean cost on FIG 2's 289 test cells.

两个阶段都训练完后，把负载调到 0.5 到 0.75 kg、延迟调到 50 到 65 ms（FIG 2 里红色左上角的边缘）：RMA 的代价大约只有鲁棒策略的一半（低于 0.65 倍）。With both phases trained, set the payload to 0.5–0.75 kg and the latency to 50–65 ms (the edge of FIG 2's red top-left corner): RMA's cost is about half the robust policy's (below 0.65 times).
打开“中途加载”，负载不超过 2 kg：看负载估计在 $t=1$ s 之后多久跟上新值，以及鲁棒策略在加载后的下垂。Turn on "pick up mid-episode" with a payload of 2 kg or less: watch how soon after $t=1$ s the payload estimate catches up with the new value, and how the robust policy sags after the pick-up.

有意思的细节Things worth noticing

(1) 在 289 个测试格子上，鲁棒策略成功 55%、平均代价 7.56；RMA 成功 74%、平均 6.13；先知成功 82%、平均 5.19。先知和鲁棒之间的差就是“知道参数”的价值，RMA 拿回了其中大部分。在标称点上 RMA 是 3.02，鲁棒是 5.24：它不必为了照顾别的环境而在这里变保守。但它也不是万能的：完全没有负载（$m=0$）且延迟 ≥ 50 ms 时，RMA 和鲁棒策略都失败，因为基础策略的增益只是 $z$ 的线性函数，连先知在那里也做不好。(2) 在 200 局随机参数上，负载估计的平均误差在第一次估计（0.1 s）后是 0.55 kg，0.55 s 时 0.17 kg，到 2 s 时 0.10 kg；延迟估计在 0.15 s 时平均误差约 4 ms，之后反而在 7 到 10 ms 之间：开头的下落和起动最能暴露延迟，关节停在目标附近以后，历史里几乎没有关于延迟的信息。估计只需要好到“动作对”就行，RMA 论文也强调 $z$ 不必在“真值”意义上正确。(3) 中途加 2 kg 时（例如 1 kg、20 ms），负载估计在 1.1 s 是 1.45 kg、1.3 s 是 2.73 kg、1.5 s 是 3.33 kg，大约 0.3 到 0.5 秒跟上，这正好是一个 0.3 s 窗口完全换成新数据所需的时间。

(1) On the 289 test cells the robust policy succeeds on 55% with a mean cost of 7.56; RMA on 74% with 6.13; the oracle on 82% with 5.19. The gap between the oracle and the robust policy is the value of knowing the parameters, and RMA recovers most of it. At the nominal point RMA costs 3.02 against the robust policy's 5.24: it need not be conservative here for the sake of other environments. It is no cure-all, though: with no payload at all ($m=0$) and latency ≥ 50 ms both RMA and the robust policy fail, because the base policy's gains are only linear in $z$, and even the oracle does poorly there. (2) Over 200 episodes with random parameters, the mean payload-estimate error is 0.55 kg after the first estimate (0.1 s), 0.17 kg at 0.55 s and 0.10 kg at 2 s; the latency estimate is off by about 4 ms on average at 0.15 s, and later by 7 to 10 ms: the initial fall and start-up reveal the delay best, and once the joint rests near the target the history contains almost no information about it. The estimate only has to be good enough for the action to be right; the RMA paper also stresses that $z$ need not be correct in a "ground-truth" sense. (3) With 2 kg added mid-episode (e.g. 1 kg, 20 ms) the payload estimate reads 1.45 kg at 1.1 s, 2.73 kg at 1.3 s and 3.33 kg at 1.5 s: it catches up within about 0.3 to 0.5 s, about the time for a 0.3 s window to fill entirely with new data.

视觉随机化：让真实图像也只是一个样本Visual randomization: make the real image just one more sample

动力学之外，另一半现实差距在传感器上，尤其是相机：渲染出的颜色、纹理、光照和噪声都和真实照片不同。12.3 节的 FIG 6 已经演示过怎样在渲染时随机化这些因素并保持标注不变。这里看它对学习的影响。任务是从机器人相机看到的一张 $12\times12$ 的彩色图里回归一个零件在桌面上的横向位置 $x$（以桌宽为单位）。网络是 $432\to32\to1$ 的 tanh MLP，每步 16 张图，训练 2000 步。

“仿真”默认渲染灰色桌面、红色零件、均匀光照、没有噪声；“真机相机”看到的是有纹理的棕色桌面、橙色零件、一侧来的光和传感器噪声，训练中从来不用。随机化强度 $s$ 从 0 到 1：桌面颜色、零件颜色、纹理、整体亮度、光照梯度和噪声都在仿真默认值和完全随机之间按 $s$ 插值。全部预测都猜桌面中央时，平均误差是 17.5%，这是“什么也没学到”的水平。

Besides dynamics, the other half of the reality gap is in the sensors, above all the camera: rendered colours, textures, lighting and noise all differ from real photographs. FIG 6 of Lesson 12.3 showed how to randomize these factors while rendering and keep the labels exact. Here we look at what that does to learning. The task: from a $12\times12$ colour image seen by the robot's camera, regress the horizontal position $x$ of a part on the table (in units of the table width). The network is a $432\to32\to1$ tanh MLP trained for 2000 steps of 16 images.

The plain simulator renders a grey table, a red part, even lighting and no noise; the "real camera" sees a textured brown table, an orange part, light from one side and sensor noise, and is never used in training. The randomization strength $s$ runs from 0 to 1: table colour, part colour, texture, overall brightness, lighting gradient and noise are interpolated by $s$ between the simulator default and fully random. Guessing the centre of the table every time gives a mean error of 17.5%, the level of "learned nothing".

FIG 6视觉域随机化：训练在随机渲染上，测试在“真机相机”上Visual domain randomization: train on random renders, test on the "real camera"

左：上排是当前随机化强度下的 4 张训练图（每 100 步换一批），下排是 4 张“真机相机”图；每张图下方的刻度条上，绿色竖线是零件的真实位置，琥珀色竖线是网络的预测。右：误差（桌宽的百分比）随训练步数的变化，琥珀色为真机相机，青色为它自己的训练分布，紫色虚线是上一次训练的真机曲线，灰色虚线是 17.5% 的“猜中央”水平；超过 60% 的值画在顶边，数值见读数。每 100 步在 200 张固定的测试图上评估一次。 Left: the top row shows 4 training images at the current strength (a new batch every 100 steps), the bottom row 4 "real camera" images; on the scale bar under each image the green tick is the part's true position and the amber tick the network's prediction. Right: error (in % of the table width) against training steps, amber for the real camera, cyan for the network's own training distribution, purple dashed the real-camera curve of the previous run, and the grey dashed line the 17.5% "guess the centre" level; values above 60% are drawn at the top edge, with the number in the readout. Every 100 steps both are evaluated on 200 fixed test images.

用 $s=0$（不随机化）训练满 2000 步：在自己的训练分布上误差只有 1.5% 左右，可在真机相机上误差超过 50%，比瞎猜还差得多。Train with $s=0$ (no randomization) for the full 2000 steps: the error on its own training distribution is about 1.5%, but on the real camera it exceeds 50%, far worse than guessing.
改用 $s=0.25$ 或 $s=0.5$ 训练满 2000 步，让真机相机误差降到 12% 以下。再试试 $s=1$：会怎样？Train with $s=0.25$ or $s=0.5$ for the full 2000 steps and get the real-camera error below 12%. Then try $s=1$: what happens?

同一个取舍，换了一种样子The same trade-off in another guise

默认种子下，$s=0$ 在真机相机上的误差约 140%（预测跑到了桌子外面：网络学会的是“找红色”，而真实图像里到处都偏红棕），$s=0.25$ 约 8%，$s=0.5$ 约 9%，$s=1$ 停在约 18%，也就是“猜中央”的水平。完全随机的颜色让零件有时和桌面几乎一样，这个只有一层隐藏层的小网络在 2000 步里根本学不会，干脆输出中央。这和 FIG 3 的结论一样：随机化要付出代价，这里的代价是学习难度，需要更大的模型（Tobin 等用的是在 ImageNet 上预训练过的 VGG-16 卷积网络）、更多数据，或者更窄、更贴近真实的范围。

With the default seed, $s=0$ is off by about 140% on the real camera (its predictions leave the table: it learned "find the red", and the real image is reddish-brown everywhere); $s=0.25$ gets about 8%, $s=0.5$ about 9%, and $s=1$ stalls near 18%, the "guess the centre" level. Fully random colours sometimes make the part nearly the colour of the table, and this small one-hidden-layer network simply cannot learn the task in 2000 steps, so it outputs the centre. It is FIG 3's lesson again: randomization has a price, here paid in learning difficulty, which calls for a bigger model (Tobin et al. used a VGG-16 convolutional network pre-trained on ImageNet), more data, or a narrower range closer to reality.

5再看数学The math

提示：把鼠标悬停在公式中带底色的符号上，对应的图元素会被高亮：$P_\phi$ 加粗 FIG 2 的训练范围，$\boldsymbol\xi^\star$ 放大 FIG 2 的紫色星并加粗 FIG 3 的紫色曲线，$J(\theta;\boldsymbol\xi_0)$ 加粗 FIG 3 的青色曲线，$\phi^L,\phi^H$ 加粗 FIG 4 的三条可动边，$\hat z_t$ 加粗 FIG 5 的琥珀色估计曲线，$d$ 加粗 FIG 1 的紫色虚线（延迟后的力矩）。Tip: hover the highlighted symbols in the formulas and the matching figure element lights up: $P_\phi$ thickens FIG 2's training range, $\boldsymbol\xi^\star$ enlarges FIG 2's purple star and thickens FIG 3's purple curve, $J(\theta;\boldsymbol\xi_0)$ thickens FIG 3's cyan curve, $\phi^L,\phi^H$ thicken FIG 4's three movable sides, $\hat z_t$ thickens FIG 5's amber estimate curves, and $d$ thickens FIG 1's purple dashed line (the delayed torque).

三个目标：专才、域随机化、最坏情况Three objectives: specialist, domain randomization, worst case

记 $J(\theta;\boldsymbol\xi)$ 为参数为 $\theta$ 的策略在环境参数 $\boldsymbol\xi$ 下一局的期望代价（20.1 节的回报取负号；本节用代价，越小越好）。三种训练目标是：

Let $J(\theta;\boldsymbol\xi)$ be the expected cost of one episode of the policy with parameters $\theta$ in the environment with parameters $\boldsymbol\xi$ (the negative of the return of Lesson 20.1; this lesson uses costs, lower is better). The three training objectives are

$$ \theta_{\text{spec}}=\arg\min_\theta J(\theta;\boldsymbol\xi_0),\qquad \theta_{\text{DR}}=\arg\min_\theta\ \mathbb E_{\boldsymbol\xi\sim\hl{box}{P_\phi}}\big[J(\theta;\boldsymbol\xi)\big],\qquad \theta_{\text{rob}}=\arg\min_\theta\ \max_{\boldsymbol\xi\in\operatorname{supp}P_\phi}J(\theta;\boldsymbol\xi). $$

专才只在标称点 $\boldsymbol\xi_0$ 上最优；域随机化最小化分布上的平均代价；最坏情况（鲁棒控制、$H_\infty$ 的思路）最小化范围里最差的那一个。域随机化不等于最坏情况：FIG 2 的“全部范围”里左上角依然失败，因为它只占平均值的一小部分。实践中用平均而不是最大，一是因为平均值可以用采样的随机梯度（20.3、20.4 节的策略梯度、6.11 节的 CEM）直接优化，最大值很难；二是最坏情况会被几乎不可能出现的极端参数绑架，过于保守。DR 的随机梯度就是普通策略梯度，只是每一局先抽一次 $\boldsymbol\xi$：

The specialist is optimal only at the nominal point $\boldsymbol\xi_0$; domain randomization minimizes the average cost over the distribution; the worst case (the robust-control or $H_\infty$ view) minimizes the worst one in the range. DR is not the worst case: in FIG 2's "full range" the top-left corner still fails, because it is only a small part of the average. In practice the average is preferred over the maximum, first because the average can be optimized directly by sampled stochastic gradients (the policy gradients of Lessons 20.3 and 20.4, the CEM of Lesson 6.11) while the maximum is hard to optimize, and second because the worst case is held hostage by extreme parameters that will almost never occur and becomes overly conservative. DR's stochastic gradient is the ordinary policy gradient with one extra draw of $\boldsymbol\xi$ per episode:

$$ \nabla_\theta\,\mathbb E_{\boldsymbol\xi\sim P_\phi}\big[J(\theta;\boldsymbol\xi)\big]=\mathbb E_{\boldsymbol\xi\sim P_\phi}\,\mathbb E_{\tau\sim p_\theta(\cdot\mid\boldsymbol\xi)}\Big[\textstyle\sum_t\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,\big(C_t(\tau)-b(s_t)\big)\Big], $$

其中 $C_t$ 是从 $t$ 起的代价和，$b$ 是基线。所以 DR 不需要新算法，它只改“环境怎样重置”。

where $C_t$ is the cost-to-go from step $t$ and $b$ a baseline. So DR needs no new algorithm; it only changes how the environment is reset.

鲁棒性的代价，与知道参数的价值The price of robustness and the value of knowing the parameters

对任何固定的 $\theta$ 和任何 $\boldsymbol\xi$ 都有 $J(\theta;\boldsymbol\xi)\ge\min_{\theta'}J(\theta';\boldsymbol\xi)$，两边取期望再对 $\theta$ 取最小，得到

For any fixed $\theta$ and any $\boldsymbol\xi$, $J(\theta;\boldsymbol\xi)\ge\min_{\theta'}J(\theta';\boldsymbol\xi)$; taking expectations and then minimizing over $\theta$ gives

$$ \min_\theta\ \mathbb E_{\boldsymbol\xi}\big[J(\theta;\boldsymbol\xi)\big]\ \ \ge\ \ \mathbb E_{\boldsymbol\xi}\Big[\min_{\theta}J(\theta;\boldsymbol\xi)\Big],\qquad J\big(\theta_{\text{DR}};\hl{nom}{\boldsymbol\xi_0}\big)\ \ge\ J\big(\theta_{\text{spec}};\boldsymbol\xi_0\big). $$

第一个不等式的左边是 DR 能做到的最好平均代价，右边是“先知”的平均代价：先知知道每一局的 $\boldsymbol\xi$，可以对每个 $\boldsymbol\xi$ 各用最好的 $\theta$。两者之差是“不知道参数”的代价，也就是把参数告诉策略能换来的好处（FIG 5 里鲁棒 7.56 对先知 5.19）；第二个不等式说在标称点上 DR 永远不比专才好（FIG 2 里 5.24 对 1.98）。两者都是对精确最优解说的；CEM 只给出近似最优，所以会有 FIG 3 里 $w=\tfrac16$ 那样几个百分点的例外。范围越宽，可选的“对所有参数都还行”的 $\theta$ 越少，两个差一般都越大，这就是 FIG 3 里基本单调上升的青色曲线。只有当策略能根据环境改变动作时，前一个差才能被缩小：这需要策略的输入里含有关于 $\boldsymbol\xi$ 的信息。

The left side of the first inequality is the best average cost DR can reach; the right side is the average cost of an "oracle" that knows each episode's $\boldsymbol\xi$ and can use the best $\theta$ for each one. Their difference is the cost of not knowing the parameters, i.e. what telling them to the policy would buy (FIG 5: robust 7.56 against oracle 5.19); the second inequality says that DR is never better than the specialist at the nominal point (FIG 2: 5.24 against 1.98). Both statements are about exact optima; CEM finds only approximate ones, hence exceptions of a few percent such as $w=\tfrac16$ in FIG 3. The wider the range, the fewer $\theta$ are "fine for every parameter", and both gaps generally grow: FIG 3's almost monotonically rising cyan curve. The first gap can only shrink if the policy can change its action with the environment, which requires its input to carry information about $\boldsymbol\xi$.

部分可观测：参数是一个看不见的状态Partial observability: the parameters are a hidden state

把 $\boldsymbol\xi$ 并进状态，$\tilde s_t=(s_t,\boldsymbol\xi)$，DR 的训练环境就是一个普通的 MDP，只是 $\boldsymbol\xi$ 这部分在一局开始时抽一次、之后不变、而且看不见：这是 20.1 节的 POMDP。POMDP 的最优策略依赖于信念 $b_t(\boldsymbol\xi)=p(\boldsymbol\xi\mid h_t)$，其中 $h_t=(s_{0:t},a_{0:t-1})$ 是历史。按贝叶斯公式

Put $\boldsymbol\xi$ into the state, $\tilde s_t=(s_t,\boldsymbol\xi)$, and DR's training environment is an ordinary MDP whose $\boldsymbol\xi$ part is drawn once at the start of the episode, never changes, and is unobserved: the POMDP of Lesson 20.1. The optimal policy of a POMDP depends on the belief $b_t(\boldsymbol\xi)=p(\boldsymbol\xi\mid h_t)$, where $h_t=(s_{0:t},a_{0:t-1})$ is the history. By Bayes' rule

$$ p(\boldsymbol\xi\mid h_t)\ \propto\ P_\phi(\boldsymbol\xi)\prod_{k<t}p(s_{k+1}\mid s_k,a_k,\boldsymbol\xi), $$

先验就是随机化分布，每一步转移都是一次“观测”。无记忆策略 $\pi(a\mid s_t)$ 只能对先验做平均（鲁棒），带记忆的策略 $\pi(a\mid h_t)$ 原则上能逼近贝叶斯最优：随着历史变长，后验收缩到真值附近，策略就趋向先知。RMA 的适应模块是一种实用的近似：它不去算完整后验，而是直接回归一个对动作有用的点估计

The prior is the randomization distribution and every transition is an "observation". A memoryless policy $\pi(a\mid s_t)$ can only average over the prior (robust); a policy with memory $\pi(a\mid h_t)$ can in principle approach the Bayes-optimal one: as the history grows the posterior contracts around the truth and the policy approaches the oracle. RMA's adaptation module is a practical approximation: instead of the full posterior it directly regresses a point estimate that is useful for acting,

$$ \hl{zhat}{\hat z_t}=\phi\big(s_{t-k:t-1},a_{t-k:t-1}\big),\qquad \min_\phi\ \mathbb E\Big[\big\|\phi(s_{t-k:t-1},a_{t-k:t-1})-\mu(e_t)\big\|^2\Big], $$

期望取在“基础策略用 $\phi$ 自己的估计”跑出的状态分布上（同策略）。均方误差的最优解是条件期望 $\mathbb E[z\mid h]$：历史不足以区分时（比如关节静止时的延迟），估计就退回到先验的均值附近，这正是 FIG 5 里延迟估计在后半局变差的原因。OpenAI 的魔方手走的是另一条路：直接用 LSTM 策略，把“推断参数”交给循环网络的隐状态，他们在论文里把这称为涌现的元学习（21.4 节的“上下文中学习”）。

with the expectation taken over the states produced when the base policy runs on $\phi$'s own estimates (on-policy). The minimizer of a mean squared error is the conditional expectation $\mathbb E[z\mid h]$: when the history cannot tell the possibilities apart (for instance the latency while the joint is at rest), the estimate falls back toward the mean of the prior, which is why FIG 5's latency estimate gets worse later in the episode. OpenAI's Rubik's-cube hand took the other route: an LSTM policy that leaves "inferring the parameters" to the recurrent hidden state, which the paper describes as emergent meta-learning (the in-context learning of Lesson 21.4).

延迟破坏马尔可夫性Latency breaks the Markov property

23.1 节第 5 节已经说明：延迟 $\hl{delay}{d}$ 下在 $t$ 时刻真正作用的力矩是 $a_{t-d/\Delta t}$，系统只在加上最近几个动作的增广状态里才是马尔可夫的，所以要把最近的动作放进观测，$o_t=(s_t,a_{t-1},\dots,a_{t-n})$。对域随机化来说这一点更要紧：延迟被随机化以后，“在路上”的动作有几个也是随机的，策略既要记住它们，又要从历史里推断延迟有多长。RMA 的基础策略输入了上一步动作 $a_{t-1}$，Peng 等的推物策略用 LSTM。本节的控制器没有动作历史，它只能靠降低增益来扛住延迟，这也是 FIG 2 左上角始终失败的根本原因之一。

Section 5 of Lesson 23.1 showed that with latency $\hl{delay}{d}$ the torque acting at time $t$ is $a_{t-d/\Delta t}$, so the system is Markov only in a state augmented with the most recent actions, and recent actions belong in the observation, $o_t=(s_t,a_{t-1},\dots,a_{t-n})$. Under domain randomization this matters even more: once the latency is random, so is the number of actions still "in flight", and the policy must both remember them and infer from history how long the delay is. RMA's base policy takes the previous action $a_{t-1}$ as input, and Peng et al.'s pushing policy used an LSTM. This lesson's controller has no action history, so its only defence against delay is lower gains, one of the root causes of FIG 2's permanently failing top-left corner.

ADR 的更新规则与熵ADR's update rule and entropy

ADR 用分解的均匀分布 $P_\phi(\boldsymbol\lambda)=\prod_{i=1}^{n}U\big(\hl{bound}{\phi_i^L},\hl{bound}{\phi_i^H}\big)$（论文记参数为 $\boldsymbol\lambda$，个数为 $d$，这里为了不和延迟混淆写作 $n$），衡量范围大小的 ADR 熵是每维平均的微分熵：

ADR uses a factorized uniform distribution $P_\phi(\boldsymbol\lambda)=\prod_{i=1}^{n}U\big(\hl{bound}{\phi_i^L},\hl{bound}{\phi_i^H}\big)$ (the paper calls the parameters $\boldsymbol\lambda$ and their number $d$; $n$ is used here to avoid a clash with the latency), and measures the size of the range by the ADR entropy, the differential entropy per dimension:

$$ \mathcal H(P_\phi)=\frac1n\sum_{i=1}^{n}\log\big(\phi_i^H-\phi_i^L\big),\qquad \phi_i^{H}\leftarrow\begin{cases}\phi_i^{H}+\Delta, & \bar p_i^{H}\ge t_H\\ \phi_i^{H}-\Delta, & \bar p_i^{H}\le t_L\\ \phi_i^{H}, & t_L<\bar p_i^{H}<t_H\end{cases} $$

$\bar p_i^H$ 是参数 $i$ 被固定在上界时最近 $M$ 局的平均成绩（论文里成绩越高越好，所以 $t_L<t_H$；FIG 4 用代价，越低越好，不等号方向反过来：代价 ≤ $t_H$ 才扩，≥ $t_L$ 就收），下界对称地向下扩、向上收。论文里 ADR 以概率 $p_b$ 做边界采样，其余时候照常从 $P_\phi$ 里抽样产生训练数据（FIG 4 简化成每次迭代固定做 3 次边界采样）。因为一个点的宽度为 0、对数是 $-\infty$，FIG 4 画熵时把每维宽度先除以全范围再以 1% 为下限；这只是画图上的约定。

$\bar p_i^H$ is the average performance of the last $M$ episodes with parameter $i$ held at its upper bound (in the paper higher performance is better, so $t_L<t_H$; FIG 4 uses a cost, lower is better, so the inequalities flip: expand if the cost is ≤ $t_H$, shrink if it is ≥ $t_L$); lower bounds expand downward and shrink upward symmetrically. In the paper ADR does boundary sampling with probability $p_b$ and otherwise samples from $P_\phi$ to generate training data as usual (FIG 4 simplifies this to 3 boundary samples per iteration). Since a point has width 0 and log-width $-\infty$, FIG 4 divides each width by the full range and floors it at 1% before taking the log; that is only a plotting convention.

没有保证，只有覆盖No guarantee, only coverage

域随机化没有一般性的迁移保证。它的论证是：如果真实参数 $\boldsymbol\xi^\star$ 落在 $P_\phi$ 的支撑里、仿真器的模型结构是对的（真正的差别只是参数）、并且训练出的策略在支撑上处处都好（而不只是平均好），那真机就只是一个训练样本。三个条件各对应一种失败：真机在范围外（FIG 2 的“不随机化”）；结构没建模（比如本节的仿真根本没有齿隙或柔性，随机化质量和延迟也补不上）；平均好但真机所在处差（FIG 2 左上角）。

Domain randomization has no general transfer guarantee. Its argument is: if the real parameters $\boldsymbol\xi^\star$ lie in the support of $P_\phi$, the simulator's model structure is right (the only real difference is in the parameters), and the trained policy is good everywhere on the support (not just on average), then the real robot is just one more training sample. Each condition has its own failure: the robot lies outside the range (FIG 2's "no randomization"); the structure is not modelled (this lesson's simulator has no backlash or flexibility at all, and randomizing mass and latency cannot make up for that); good on average but poor where the robot is (FIG 2's top-left corner).

6机器人中的它In robotics

真实系统里In real systems

视觉：Tobin、Fong、Ray、Schneider、Zaremba 与 Abbeel（IROS 2017）用 MuJoCo 的渲染器生成随机纹理、随机光照、随机相机位置、0 到 10 个干扰物的低保真图像，只用这些图训练物体定位网络（VGG-16，ImageNet 预训练），在真实图像上定位误差约 1.5 cm，并在 Fetch 机器人上用于杂乱环境中的抓取。同年 Sadeghi 与 Levine 的 CAD2RL（RSS 2017）用大量随机化渲染场景训练无人机避障策略，标题就是“不用一张真实图像”。 动力学：Peng、Andrychowicz、Zaremba 与 Abbeel（ICRA 2018）在 7 自由度 Fetch 机械臂的推冰球任务上随机化了 95 个参数（各连杆质量、关节阻尼、冰球的质量摩擦与阻尼、桌面高度、位置控制器增益、动作之间的时间步长、观测噪声），LSTM 策略在真机上的成功率是 0.89，而没有随机化的前馈网络是 0；去掉“动作时间步长”的随机化（即不建模延迟）后成功率掉到 0.29。 足式机器人：Tan 等（RSS 2018）在 Minitaur 四足上结合改进的执行器模型、延迟建模和动力学随机化把仿真步态迁移到真机；后来的 legged_gym / Isaac Lab 类配置通常随机化地面摩擦、附加的机身质量，并随机推机器人。 灵巧手：OpenAI 的 Dactyl（Andrychowicz 等，IJRR 2020）在大量随机化的仿真里训练 LSTM 策略转方块；“用机械手解魔方”（OpenAI，2019）提出了 ADR，并在 Shadow 灵巧手上完成需要 15 次面旋转的魔方 60% 的时间、需要 26 次的 20% 的时间（魔方的解法由 Kociemba 求解器给出，手负责执行）；论文里策略还扛住了训练时没见过的扰动，比如戴橡胶手套、绑住手指、用毛绒长颈鹿戳魔方。 自适应：Yu、Tan、Liu 与 Turk（RSS 2017）的 UP-OSI 先训练以动力学参数为输入的“通用策略”，再训练从历史估计参数的在线辨识网络，是 RMA 的前身；RMA（Kumar 等，RSS 2021）在 RaiSim 里训练 Unitree A1，环境向量 $e_t$ 有 17 维（附加质量及其在机身上的位置 3 维、12 个电机的强度、摩擦、局部地形高度），编码成 8 维的 $z_t$；适应模块读最近 50 步（0.5 s）的状态和动作，以 10 Hz 运行，基础策略以 100 Hz 运行，部署到真机时没有任何微调。训练时负载随机化在 0 到 6 kg，测试时用到 7 kg，测试范围有意比训练范围宽。 用真实数据调范围：SimOpt（Chebotar 等，ICRA 2019）用少量真机轨迹去更新随机化分布本身，让它更贴近真机；Active Domain Randomization（Mehta 等，CoRL 2019）主动去找策略最差的参数区域多采样。

Vision: Tobin, Fong, Ray, Schneider, Zaremba and Abbeel (IROS 2017) rendered low-fidelity images in MuJoCo with random textures, random lighting, random camera positions and 0 to 10 distractor objects, trained an object-localization network (VGG-16, pre-trained on ImageNet) on these images only, and reached about 1.5 cm localization error on real images, good enough for grasping in clutter with a Fetch robot. In the same year Sadeghi and Levine's CAD2RL (RSS 2017) trained a drone collision-avoidance policy on many randomized rendered scenes; its title says "without a single real image". Dynamics: Peng, Andrychowicz, Zaremba and Abbeel (ICRA 2018) randomized 95 parameters for puck pushing with a 7-DoF Fetch arm (every link mass, joint damping, the puck's mass, friction and damping, table height, position-controller gains, the time step between actions, observation noise); their LSTM policy succeeded 0.89 of the time on the real robot, against 0 for a feedforward network trained without randomization, and removing the randomization of the action time step (i.e. not modelling latency) dropped success to 0.29. Legged robots: Tan et al. (RSS 2018) combined an improved actuator model, latency modelling and dynamics randomization to transfer gaits from simulation to the Minitaur quadruped; later legged_gym / Isaac Lab style configurations commonly randomize ground friction and added base mass and push the robot at random. Dexterous hands: OpenAI's Dactyl (Andrychowicz et al., IJRR 2020) trained an LSTM policy for block reorientation in heavily randomized simulation; "Solving Rubik's Cube with a Robot Hand" (OpenAI, 2019) introduced ADR and, on a Shadow Dexterous Hand, completed cubes requiring 15 face rotations 60% of the time and ones requiring 26 face rotations 20% of the time (the solution sequence comes from the Kociemba solver; the hand executes it); the paper also shows the policy coping with perturbations it never trained on, such as a rubber glove, tied fingers and a plush giraffe poking the cube. Adaptation: UP-OSI by Yu, Tan, Liu and Turk (RSS 2017) first trains a "universal policy" that takes the dynamics parameters as input and then an online system-identification network that estimates them from history, a forerunner of RMA. RMA (Kumar et al., RSS 2021) trained a Unitree A1 in RaiSim with a 17-dimensional environment vector $e_t$ (the added mass and its position on the body, 3; the strengths of the 12 motors; friction; local terrain height) encoded into an 8-dimensional $z_t$; the adaptation module reads the last 50 steps (0.5 s) of states and actions at 10 Hz while the base policy runs at 100 Hz, and it was deployed on the robot without any fine-tuning. The payload was randomized over 0 to 6 kg in training and up to 7 kg in testing: the test ranges were deliberately wider than the training ranges. Tuning the range with real data: SimOpt (Chebotar et al., ICRA 2019) updates the randomization distribution itself from a few real trajectories so that it matches the robot better; Active Domain Randomization (Mehta et al., CoRL 2019) actively seeks out the parameter regions where the policy is worst and samples them more.

一份随机化清单。实际项目里通常随机化：(1) 惯性参数：连杆和负载的质量、质心，误差一般给测量值的 ±10% 到 ±20% 量级，负载按任务给全范围；(2) 接触：摩擦系数、恢复系数、地形；(3) 执行器：电机强度（力矩常数）、PD 增益、延迟和控制周期抖动、关节阻尼与摩擦；(4) 传感器：观测噪声和偏置、IMU 漂移、相机内外参的小扰动；(5) 外界：随机推力、随机初始状态；(6) 视觉：纹理、颜色、光照、相机位姿、干扰物。范围怎么定比列表更重要：能测的先测（23.1 节的系统辨识），用测量的不确定度定范围；测不了的给宽范围，再看 FIG 3 那样的曲线检查是不是太宽。

A randomization checklist. Real projects typically randomize: (1) inertial parameters: link and payload masses and centres of mass, with errors of order ±10% to ±20% of the measurement, and the payload over the task's full range; (2) contact: friction coefficients, restitution, terrain; (3) actuators: motor strength (torque constant), PD gains, latency and control-period jitter, joint damping and friction; (4) sensors: observation noise and bias, IMU drift, small perturbations of camera intrinsics and extrinsics; (5) the outside world: random pushes, random initial states; (6) vision: textures, colours, lighting, camera pose, distractors. How the ranges are set matters more than the list: measure what you can first (system identification, Lesson 23.1) and size the range by the measurement's uncertainty; give unmeasurable parameters a wide range, then check with a curve like FIG 3 that it is not too wide.

7常见误区Common pitfalls

误区 1：范围越宽越安全Pitfall 1: the wider the range, the safer

FIG 3 和 FIG 6 都说明不是。范围过宽有两种代价：策略为了照顾不可能出现的极端参数而变保守（标称代价从 1.98 涨到 5.24，覆盖率反而从 59% 掉到 55%）；学习本身变难，容量不够的网络干脆什么也学不会（FIG 6 的 $s=1$）。还有一种隐蔽的情况：范围宽到包含了任务根本不可解的参数，奖励信号变得很嘈杂，训练变慢甚至不收敛。ADR 的“从一个点开始、能做到才扩大”就是为了绕开这一点。

FIG 3 and FIG 6 both say no. An over-wide range costs in two ways: the policy turns conservative to cope with extreme parameters that will never occur (the nominal cost rises from 1.98 to 5.24 while coverage drops from 59% to 55%), and learning itself gets harder, so a network without enough capacity learns nothing at all (FIG 6 at $s=1$). There is a subtler case too: a range wide enough to include parameters where the task is impossible makes the reward signal noisy and slows or stalls training. ADR's "start from a point and expand only what has been mastered" is designed to avoid exactly this.

误区 2：随机化了很多东西，却漏了真正的差距Pitfall 2: randomizing many things but missing the real gap

随机化只能覆盖仿真里有的参数。如果真机的问题是延迟，而仿真根本没有延迟这个机制，把质量和摩擦随机化得再宽也没用：Peng 等的消融里，只是把动作时间步长固定，真机成功率就从 0.89 掉到 0.29。先用少量真机数据找出差距在哪里（对比仿真与真机对同一串动作的响应，再用 23.1 节的系统辨识去拟合），再决定随机化什么；常被漏掉的是延迟、控制周期抖动、执行器带宽和饱和、传感器滤波。

Randomization can only cover parameters the simulator has. If the real problem is latency and the simulator has no latency mechanism at all, randomizing mass and friction however widely will not help: in Peng et al.'s ablation, merely fixing the action time step dropped real-robot success from 0.89 to 0.29. Use a little real data to find where the gap is (compare how the simulator and the robot respond to the same action sequence, then fit it with the system identification of Lesson 23.1) before deciding what to randomize; often forgotten are latency, control-period jitter, actuator bandwidth and saturation, and sensor filtering.

误区 3：抽样方式不符合物理Pitfall 3: sampling that violates physics

独立地均匀抽每个参数，可能抽出不物理的组合（负质量、质心跑到机身外、惯量矩阵不满足三角不等式），也会让相关的参数失去相关性。尺度型参数（质量、阻尼、摩擦）常用对数均匀抽样，Peng 等就是这样做的；该按局抽的（负载）不要按步抽，该按步变的（控制周期抖动、观测噪声）不要整局固定：两者对策略的要求完全不同。

Drawing every parameter independently and uniformly can produce unphysical combinations (negative mass, a centre of mass outside the body, inertia matrices that violate the triangle inequality) and strips correlated parameters of their correlation. Scale-type parameters (masses, damping, friction) are often sampled log-uniformly, as Peng et al. did; what should be drawn per episode (payload) should not be redrawn every step, and what changes every step (control-period jitter, observation noise) should not be frozen for the whole episode: the two demand completely different things of the policy.

误区 4：用无记忆策略去扛很宽的范围Pitfall 4: covering a very wide range with a memoryless policy

无记忆策略只能给出对所有参数都“还行”的折中，范围越宽，折中越差。如果参数在一局里基本不变且影响很大（负载、地面），给策略历史（帧堆叠、动作历史、循环网络或 RMA 式的适应模块）往往比继续调范围有效得多：FIG 5 里成功比例从 55% 升到 74%。同时别忘了延迟本身就要求把最近的动作放进观测。

A memoryless policy can only offer a compromise that is "fine" for every parameter, and the wider the range, the worse the compromise. When the parameters are roughly constant within an episode and matter a lot (payload, ground), giving the policy history (frame stacking, action history, a recurrent network, or an RMA-style adaptation module) is often far more effective than further tuning of the range: in FIG 5 the success fraction rises from 55% to 74%. And do not forget that latency on its own calls for putting recent actions in the observation.

误区 5：只在训练分布里评估Pitfall 5: evaluating only inside the training distribution

在训练范围里测出的成功率说明不了迁移，因为真机多半不正好在你以为的地方。像 FIG 2 那样在训练范围之外也测一张图，或者像 RMA 那样有意用比训练更宽的测试范围，才能看出性能在边界外是缓慢下降还是悬崖式崩溃。悬崖（比如延迟一超过某个值就振荡）意味着真机只要稍有不同就会出事，应该把范围往那个方向多放一点，或者换一个对它不敏感的策略结构。

A success rate measured inside the training range says nothing about transfer, because the real robot is probably not exactly where you think. Testing a map beyond the training range as FIG 2 does, or deliberately testing on wider ranges than training as RMA did, shows whether performance degrades gently past the boundary or falls off a cliff. A cliff (oscillation as soon as the latency exceeds some value, say) means a slightly different robot will fail: widen the range in that direction, or change to a policy structure that is insensitive to it.

8小练习Check yourself

1. 域随机化的训练目标是什么？1. What objective does domain randomization optimize?

DR 最小化 $\mathbb E_{\boldsymbol\xi\sim P_\phi}[J(\theta;\boldsymbol\xi)]$，所以选项 C 对。选项 A 是专才，选项 B 是最坏情况（鲁棒控制）的目标，DR 不保证最坏情况好：FIG 2 全部范围训练后左上角依然失败。选项 D 是系统辨识的目标，它和 DR 互补而不是同一回事。DR minimizes $\mathbb E_{\boldsymbol\xi\sim P_\phi}[J(\theta;\boldsymbol\xi)]$, so option C. Option A is the specialist; option B is the worst-case (robust-control) objective, and DR does not promise a good worst case: after full-range training in FIG 2 the top-left corner still fails. Option D is the goal of system identification, which complements DR rather than being the same thing.

2. 把训练范围从一个点放宽到全部范围后，标称参数上的代价从 1.98 变成 5.24。最好的解释是：2. Widening the training range from a point to the full range raises the cost at the nominal parameters from 1.98 to 5.24. The best explanation is:

选项 B。专才 $\theta_{\text{spec}}$ 按定义在 $\boldsymbol\xi_0$ 上最优，任何别的 $\theta$ 在那里都不会更好；DR 选了一个对大延迟也稳定的低增益（$K_p$ 从约 122 降到约 37），在没有延迟的标称点上就跟得慢。这不是没收敛（选项 A）；DR 也没让所有环境变差，真机就从 15.70 降到 4.95（选项 C 错）；标称点 $(1,0)$ 在 $[0,4]\times[0,80]$ 里（选项 D 错）。Option B. The specialist $\theta_{\text{spec}}$ is by definition optimal at $\boldsymbol\xi_0$, so no other $\theta$ does better there; DR picked low gains that stay stable under long delays ($K_p$ from about 122 down to about 37), which track slowly at the delay-free nominal point. It is not a lack of convergence (option A); DR did not make every environment worse, since the real robot went from 15.70 to 4.95 (option C is wrong); and the nominal point $(1,0)$ lies inside $[0,4]\times[0,80]$ (option D is wrong).

3. 在 ADR 中，负载上界 $\phi^H_m$ 什么时候会向外扩大一步？3. In ADR, when does the payload's upper bound $\phi^H_m$ move outward by one step?

选项 D：这就是“边界采样”，每条边有自己的缓冲区，只看这条边上的表现。选项 A 是固定课程，不是 ADR；ADR 完全在仿真里运行，看不到真机（选项 B）；用整个范围的平均（选项 C）无法区分是哪一条边太难，正因如此 ADR 才逐条边测试。Option D: this is "boundary sampling"; each bound has its own buffer and is judged only by performance at that bound. Option A is a fixed curriculum, not ADR; ADR runs entirely in simulation and never sees the real robot (option B); an average over the whole range (option C) cannot tell which bound is too hard, which is exactly why ADR tests bound by bound.

4. RMA 的适应模块 $\phi$ 是怎样训练的？4. How is RMA's adaptation module $\phi$ trained?

选项 A。RMA 的第 2 阶段是监督学习，目标是 $z_t$ 而不是 $e_t$：只需估出“动作该怎样变”所需的信息。数据是同策略的，因为只用基础策略拿真 $z$ 跑出的完美轨迹训练，$\phi$ 就没见过自己估错时的状态（选项 C 的问题）。RMA 部署时不做任何微调（选项 B 错）；第 1 阶段用 PPO 训练的是基础策略和编码器 $\mu$，不是 $\phi$（选项 D 错）。Option A. RMA's phase 2 is supervised learning with target $z_t$, not $e_t$: it only has to estimate what is needed to change the action. The data is on-policy, because training only on perfect trajectories of the base policy run with the true $z$ would never show $\phi$ the states that follow its own mistakes (the flaw of option C). RMA is deployed without any fine-tuning (option B is wrong); phase 1 trains the base policy and the encoder $\mu$ with PPO, not $\phi$ (option D is wrong).

5. 仿真里随机化了 0 到 60 ms 的执行器延迟，策略只输入当前的关节角和角速度。最值得做的改进是：5. Actuator latency is randomized over 0 to 60 ms in simulation, and the policy sees only the current joint angle and velocity. The most worthwhile improvement is:

选项 C。有延迟时，下一状态取决于还“在路上”的旧动作，只看 $(q,\dot q)$ 的策略无从得知它们，只能靠低增益硬扛；放进动作历史后，策略既能补偿在途的力矩，也能从历史推断延迟本身（RMA 的基础策略就输入了 $a_{t-1}$）。选项 A 只会让策略更保守；选项 B 去掉了对延迟抖动和测量误差的鲁棒性，而测量本身值得做，但它不解决非马尔可夫的问题；延迟以毫秒计，与控制频率无关，提高频率只是让它跨越更多步（选项 D 错）。Option C. With latency the next state depends on old actions still "in flight"; a policy that sees only $(q,\dot q)$ cannot know them and must fall back on low gains. With the action history it can compensate for the torque in flight and also infer the latency itself from history (RMA's base policy takes $a_{t-1}$ as input). Option A only makes the policy more conservative; option B gives up robustness to latency jitter and measurement error, and while measuring is worthwhile it does not fix the non-Markov problem; latency is measured in milliseconds and does not depend on the control rate: a faster rate merely spreads it over more steps (option D is wrong).

9延伸阅读Further reading

J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba & P. Abbeel, Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World, IROS 2017 — 视觉域随机化的开山之作：只用随机纹理渲染训练检测器the founding paper of visual domain randomization: detectors trained only on randomly textured renders
F. Sadeghi & S. Levine, CAD2RL: Real Single-Image Flight without a Single Real Image, RSS 2017 — 随机化渲染场景训练无人机避障drone collision avoidance trained on randomized rendered scenes
X. B. Peng, M. Andrychowicz, W. Zaremba & P. Abbeel, Sim-to-Real Transfer of Robotic Control with Dynamics Randomization, ICRA 2018 — 动力学随机化与 LSTM 策略，含延迟随机化的消融dynamics randomization with an LSTM policy, including the latency ablation
J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez & V. Vanhoucke, Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, RSS 2018 — 执行器建模加随机化，把四足步态迁移到真机actuator modelling plus randomization to move quadruped gaits to hardware
W. Yu, J. Tan, C. K. Liu & G. Turk, Preparing for the Unknown: Learning a Universal Policy with Online System Identification, RSS 2017 — 以参数为输入的通用策略加在线辨识，RMA 的前身a parameter-conditioned universal policy plus online identification, a forerunner of RMA
OpenAI: M. Andrychowicz et al., Learning Dexterous In-Hand Manipulation, IJRR 2020 — Dactyl：大规模随机化与 LSTM 策略的灵巧手Dactyl: a dexterous hand from massive randomization and an LSTM policy
OpenAI: I. Akkaya et al., Solving Rubik's Cube with a Robot Hand, arXiv:1910.07113, 2019 — 自动域随机化（ADR）与涌现的元学习automatic domain randomization (ADR) and emergent meta-learning
A. Kumar, Z. Fu, D. Pathak & J. Malik, RMA: Rapid Motor Adaptation for Legged Robots, RSS 2021 — 特权基础策略加从历史估计 extrinsics 的适应模块a privileged base policy plus an adaptation module that estimates the extrinsics from history
Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff & D. Fox, Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience, ICRA 2019 — SimOpt：用真机数据调整随机化分布SimOpt: adjusting the randomization distribution with real-robot data
B. Mehta, M. Diaz, F. Golemo, C. J. Pal & L. Paull, Active Domain Randomization, CoRL 2019 — 主动寻找最难的参数区域actively seeking the hardest parameter regions