Part VI · 第 22 章模仿学习与数据 · 22.3Part VI · Ch. 22 Imitation Learning & Data · 22.3

遥操作与数据采集：ALOHA、UMITeleoperation & Data Collection: ALOHA, UMI

模仿学习（22.1–22.2 节）留下了一个问题：示范数据从哪里来？答案几乎总是：一个人通过某种接口操纵机器人或它的替身，同时把它看到的和做的都记录下来。这一节从接口讲起：主从臂为什么直接复制关节角、不解逆运动学；延迟怎样拖慢操作员；UMI 的手持夹爪为什么用腕部相机和相对动作，又为什么要做延迟匹配；以及在采集预算有限时，为什么多换几个环境比在同一个环境里多录几条更值。 Imitation learning (Lessons 22.1–22.2) leaves one question open: where do the demonstrations come from? Almost always from a person driving a robot, or a stand-in for it, through some interface while everything it sees and does is recorded. This lesson starts from the interfaces: why leader–follower arms copy joint angles instead of solving inverse kinematics; how latency slows the operator down; why UMI's hand-held gripper uses a wrist camera and relative actions, and why it matches latencies; and why, on a fixed collection budget, more environments are worth more than more demonstrations in the same one.

约 40 分钟~40 min 前置：22.1 BC 与 DAgger · 7.5 逆运动学 · 3.4 机器人雅可比 · 5.3 SE(3) · 9.4 执行器与传感器 · 16.2 泛化Prereqs: 22.1 BC & DAgger · 7.5 Inverse kinematics · 3.4 Robot Jacobian · 5.3 SE(3) · 9.4 Actuators & sensors · 16.2 Generalization 5 个交互5 interactives

1为什么机器人需要它Why robots need this

22.1 节的行为克隆把学策略变成了监督学习：给它成对的（观测，动作），它学会从前者预测后者。这个配方里最贵的不是 GPU，而是这些成对的数据。图片和文字可以从网上爬，机器人数据却要有人当场做一遍。看看量级：ALOHA 论文里每个真实任务录 50 条示范（穿魔术贴的任务录了 100 条），每条只有 8–14 秒，可是算上复位，一个任务要 30–60 分钟墙钟时间，平均每条 36–72 秒；UMI 在 30 个地点录的 1400 条示范用了 12 个人时，平均每条约 31 秒。一条示范要花几十秒到一分钟以上的人工，而一个任务通常只有几十到几百条。

所以一个采集系统要回答四个问题：

映射：人的动作怎样变成机器人的动作？主从臂、VR 手柄、手机、空间鼠标、外骨骼、手持夹爪，各有各的映射方式和代价（FIG 1）。
闭环：操作员看到的是多久以前的画面？延迟会让人变慢、变抖，而慢而抖的示范也会被策略学去（FIG 2）。
记录：记下什么，在哪个坐标系里，带什么时间戳？这决定了数据能不能换一台机器人、换一个地点继续用（FIG 3、FIG 4）。
录多少、在哪里录：预算有限时，是在同一张桌子上多录几条，还是多换几个环境（FIG 5）？

这一节不重复 22.1–22.2 的学习算法，只关心喂给它们的数据。

The behaviour cloning of Lesson 22.1 turns learning a policy into supervised learning: give it (observation, action) pairs and it learns to predict the second from the first. The expensive ingredient in that recipe is not the GPU but the pairs. Images and text can be scraped from the web; robot data has to be performed by someone, on the spot. Look at the scale: the ALOHA paper records 50 demonstrations per real-world task (100 for the Velcro-threading task), each only 8–14 s long, yet with resets a task takes 30–60 minutes of wall-clock time, 36–72 s per demonstration on average; UMI's 1400 demonstrations in 30 locations took 12 person-hours, about 31 s each. One demonstration costs tens of seconds to over a minute of a person's time, and a task usually gets only tens to hundreds of them.

A collection system therefore has to answer four questions:

Mapping: how does the person's motion become the robot's motion? Leader–follower arms, VR controllers, phones, space mice, exoskeletons and hand-held grippers each map differently and pay different prices (FIG 1).
Closing the loop: how old is the picture the operator acts on? Latency makes people slow and shaky, and the policy learns slow, shaky demonstrations too (FIG 2).
Recording: what is recorded, in which coordinate frame, with which timestamps? That decides whether the data still works on another robot or in another place (FIG 3, FIG 4).
How much, and where: on a fixed budget, do you record more demonstrations at the same table or move to more environments (FIG 5)?

This lesson does not repeat the learning algorithms of Lessons 22.1–22.2; it is about the data that feeds them.

2直觉：主从臂复制的是关节，不是位姿Intuition: a leader arm copies joints, not poses

最直接的遥操作接口是给操作员一台"小号的机器人"。ALOHA 用两台小的 WidowX 作为主臂（leader），两台 6 自由度的 ViperX 作为从臂（follower），都来自 Trossen Robotics。操作员握住主臂的手柄拖动它，从臂以 50 Hz 复制主臂的关节角，主臂手柄上加装的"剪刀式"机构连续控制夹爪开合。整套系统（含相机等附件）在 2 万美元以内。GELLO 把这个想法推到更便宜：用 3D 打印件和 Dynamixel XL330 舵机做出目标机械臂的运动学等价缩小版（连杆长度按 DH 参数缩放到 0.5 倍），每台物料成本不到 300 美元，支持 Franka、UR5 和 xArm。

为什么复制关节角，而不是把主臂手爪的位姿交给从臂、让从臂自己解逆运动学（7.5 节）？因为只要主臂和从臂的关节结构相同、连杆比例相同，"关节角相同"就意味着"手爪位姿相同（差一个缩放）"，根本不需要 IK：没有奇异位形时的数值问题，没有多解时"肘部翻转"的问题，主臂能摆出的每一个姿态从臂都摆得出。ALOHA 论文明确说，现成的 IK 在奇异位形附近经常失败，所以选了关节空间映射。

代价是：主臂必须是从臂的"运动学复制品"。VR 手柄、手机、空间鼠标、手部追踪给出的是位姿，没有关节；它们只能走"任务空间重定向"：把人手的位姿（经过缩放、平移）当成从臂手爪的目标，每一帧解一次 IK。这种方式对任意机械臂都适用，却继承了 IK 的全部麻烦：目标超出从臂的工作空间、靠近奇异位形、关节限位、多解之间跳变。

下图左边是主臂（拖动关节圆圈），右边是从臂。先用"半尺寸复制品"主臂和"复制关节"映射动一动，再换成"比例不同"的主臂，最后切到"任务空间 IK"。

The most direct teleoperation interface hands the operator a "small copy of the robot". ALOHA uses two small WidowX arms as leaders and two 6-DoF ViperX arms as followers, all from Trossen Robotics. The operator grabs the leader's handle and moves it; the follower copies the leader's joint angles at 50 Hz, and a "handle and scissor" mechanism on the leader controls the gripper continuously. The whole system, cameras and other add-ons included, stays within $20k. GELLO pushes the idea further down in price: 3D-printed parts and Dynamixel XL330 servos form a kinematically equivalent miniature of the target arm (link lengths scaled by 0.5 from its DH parameters), with a bill of materials under $300 per device, for the Franka, the UR5 and the xArm.

Why copy joint angles instead of handing the leader's tool pose to the follower and letting it solve inverse kinematics (Lesson 7.5)? Because as long as leader and follower share the same joint structure and the same link proportions, "same joint angles" means "same tool pose, up to a scale", and no IK is needed: no numerical trouble at singular postures, no "elbow flips" between multiple solutions, and every posture the leader can take the follower can take too. The ALOHA paper says explicitly that off-the-shelf IK often fails near singularities, which is why it chose joint-space mapping.

The price: the leader must be a kinematic replica of the follower. VR controllers, phones, space mice and hand tracking give a pose, not joints; they can only use "task-space retargeting": take the hand's pose (scaled and shifted) as the target for the follower's tool and solve IK every frame. That works for any arm but inherits all of IK's trouble: targets outside the follower's workspace, singular postures, joint limits, and jumps between solutions.

In the figure the leader is on the left (drag the joint circles) and the follower on the right. Move the "half-scale replica" leader with the "copy joints" mapping first, then switch to the "different proportions" leader, and finally to "task-space IK".

FIG 1主从映射：复制关节 vs 任务空间 IKLeader–follower mapping: copy joints vs task-space IK

主臂：拖动关节圆圈LEADER · drag the joint circles

从臂（尺寸 ×2 左右）FOLLOWER (about ×2 in size)

从臂是平面 3R 臂，连杆 0.40 / 0.32 / 0.12 m（虚线圆是它的可达半径 0.84 m）。主臂可选"半尺寸复制品"（0.20 / 0.16 / 0.06 m）或"比例不同"（0.13 / 0.21 / 0.06 m）。空心琥珀色标记是操作员想要的手爪位姿：主臂手爪位置除以运动缩放 $s$，朝向不变；实心琥珀点是从臂手爪的实际位置；红色线段是两者的位置误差。关节用红、绿、蓝圆圈标出（基座、肘、腕）。任务空间 IK 用阻尼最小二乘（$\lambda=0.04$，朝向一行权重 0.15 m/rad），每次更新迭代 20 次，从上一次的解热启动。 The follower is a planar 3R arm with links 0.40 / 0.32 / 0.12 m (the dashed circle is its reach, 0.84 m). The leader is either a "half-scale replica" (0.20 / 0.16 / 0.06 m) or has "different proportions" (0.13 / 0.21 / 0.06 m). The hollow amber marker is the tool pose the operator means: the leader's tool position divided by the motion scale $s$, same orientation; the filled amber dot is where the follower's tool actually is; the red segment is the position error between them. The joints are marked with red, green and blue circles (base, elbow, wrist). Task-space IK uses damped least squares ($\lambda=0.04$, orientation row weighted 0.15 m/rad) with 20 iterations per update, warm-started from the previous solution.

试一试Try this

保持"复制关节"，换成"比例不同"的主臂，把它摆到一个弯曲的姿态，使从臂手爪偏离想要的位置超过 10 cm。为什么手臂伸直时误差最小（只剩 4 cm）？Keep "copy joints", switch to the "different proportions" leader and bend it until the follower's tool is more than 10 cm from where the operator means it to be. Why is the error smallest (only 4 cm) when the arm is straight?
还是"比例不同"的主臂，切到"任务空间 IK"，保持主臂肘部明显弯曲（$|q_2|>0.6$ rad）：从臂手爪的位置误差小于 5 mm、朝向误差小于 2°。这时两条臂的关节角还一样吗？With the same "different proportions" leader, switch to "task-space IK" and keep the leader's elbow clearly bent ($|q_2|>0.6$ rad): the follower's tool is within 5 mm and 2° of the target. Are the two arms' joint angles still the same?
在"任务空间 IK"下把运动缩放 $s$ 调小（放大人的动作），再把主臂伸展开，让目标落到从臂够不到的地方：IK 的残差超过 5 cm，从臂被拉直。"复制关节"模式下缩放有作用吗？In "task-space IK", lower the motion scale $s$ (amplifying the operator's motion) and stretch the leader out until the target lies beyond the follower's reach: the IK residual exceeds 5 cm and the follower is pulled straight. Does the scale do anything in "copy joints" mode?

核心洞见Key insight

主从臂之所以简单可靠，是因为它把"解 IK"这件事交给了操作员的手：人直接摆出关节构型，从臂照抄。只要两条臂运动学相似，关节空间的复制就同时保证了位姿一致、没有奇异和多解问题。任务空间重定向换来的是通用性（任意接口、任意机械臂），付出的是 IK 的全部麻烦，以及"人想要的位姿根本到不了"的情况。

Leader–follower arms are simple and reliable because they hand "solving IK" to the operator's hand: the person forms the joint configuration directly and the follower copies it. As long as the two arms are kinematically similar, copying in joint space guarantees matching poses with no singularities or multiple solutions. Task-space retargeting buys generality (any interface, any arm) and pays with all of IK's trouble, plus the cases where the pose the person wants simply cannot be reached.

3动手玩：延迟让操作员变慢Play: latency slows the operator down

遥操作时，操作员本身就是控制器：看到手爪离目标还有多远，就按比例把手往那个方向移。这个回路里有两段延迟：人自己的反应时间（这里取 150 ms），和系统延迟（指令传到从臂 + 视频传回操作员，网络、编码、显示都在里面）。9.4 节在关节伺服回路里看过：延迟不改变增益，只吃掉相位，增益太大就会振荡。人也一样。延迟大时，人会本能地放慢，或者"动一下、等一下"（Ferrell 在 1965 年就观察到了这种 move-and-wait 策略）。

主从臂还有一个常被忽视的好处：操作员能用自己的手感觉主臂在哪里（本体感觉），这条局部回路没有系统延迟。主臂与从臂运动学匹配时，手在哪里，从臂过一会儿就到哪里，所以在空中移动时，视频延迟没那么要紧。但只能从视频里看到的东西（移动的物体、接触是否发生）仍然是迟到的。下图里，操作员要把从臂手爪移到 20 cm 外的目标，±5 mm 内停住 0.3 s 才算完成。

During teleoperation the operator is the controller: seeing how far the gripper still is from the target, they move their hand that way in proportion. The loop has two delays: the person's own reaction time (150 ms here) and the system latency (the command travelling to the follower plus the video travelling back, with network, encoding and display all included). Lesson 9.4 showed it for a joint servo: delay leaves the gain alone and eats phase, and too much gain makes the loop ring. People are no different. With a large delay they instinctively slow down, or "move a little, then wait" (Ferrell observed this move-and-wait strategy as early as 1965).

Leader–follower arms have one more, often overlooked, advantage: operators feel where the leader is with their own hand (proprioception), and that local loop has no system latency. When leader and follower are kinematically matched, wherever the hand is, the follower will be a moment later, so for motion through free space the video delay matters less. But anything seen only in the video (a moving object, whether contact has happened) still arrives late. In the figure the operator must bring the follower's gripper to a target 20 cm away; it counts as done when it stays within ±5 mm for 0.3 s.

FIG 2带延迟的人在回路A human in a delayed loop

上：操作员一侧 · 下：机器人一侧TOP: operator side · BOTTOM: robot side

位置 (m) vs 时间 (s)POSITION (m) vs TIME (s)

琥珀色 = 从臂手爪 $x_f$，紫色 = 主臂 / 操作员的手 $x_\ell$，绿色虚线与绿带 = 目标及 ±5 mm 容差；上方的空心琥珀标记是操作员此刻在视频里看到的从臂（晚了视频延迟）。操作员模型：手速 $=K\times$ 感知到的误差，限幅 0.6 m/s，反应延迟 150 ms；系统延迟一半在指令通路、一半在视频通路；从臂伺服的时间常数 30 ms。"自己的手"模式下，操作员按主臂的位置闭环，目标仍从视频里看。仿真步长 2 ms，共 6 s，播放速度与真实时间一致。 Amber = the follower's gripper $x_f$, purple = the leader / operator's hand $x_\ell$, green dashed line and band = the target and its ±5 mm tolerance; the hollow amber marker on top is the follower as the operator sees it in the video right now (late by the video delay). Operator model: hand speed $=K\times$ perceived error, limited to 0.6 m/s, with a 150 ms reaction delay; the system latency is split equally between the command path and the video path; the follower's servo has a 30 ms time constant. In "own hand" mode the operator closes the loop on the leader's position and still sees the target through the video. The simulation uses 2 ms steps over 6 s and plays back in real time.

试一试Try this

看视频闭环，把系统延迟调到 300 ms 以上：默认增益下手爪来回振荡，6 s 内停不下来。降低增益 $K$，让它在 2.0 s 内完成。最快能多快？Watching the video, raise the system latency to 300 ms or more: at the default gain the gripper swings back and forth and does not settle within 6 s. Lower the gain $K$ until it finishes within 2.0 s. How fast can it get?
保持延迟 ≥ 300 ms，换成"自己的手"闭环，增益 $K\ge3$：1.0 s 内完成。再把目标换成"移动"：这时延迟还要紧吗？Keep the latency at 300 ms or more, switch to "own hand" feedback with gain $K\ge3$: it finishes within 1.0 s. Then switch the target to "moving": does latency still matter?

核心洞见Key insight

延迟给操作员的增益设了一个上限，大约是 $\pi/(2\tau)$，$\tau$ 是整个回路的延迟。增益受限，最快的完成时间就和延迟成正比：无延迟时最快 0.60 s，300 ms 时最快 1.50 s。示范因此变慢、变犹豫，策略学到的也是这样的动作。降低延迟最有效的办法，是给操作员一条不经过网络的局部回路：主从臂的本体感觉、VR 里本地渲染的手。

Latency caps the operator's gain at about $\pi/(2\tau)$, where $\tau$ is the delay around the whole loop. With the gain capped, the fastest completion time grows in proportion to the delay: 0.60 s at best without latency, 1.50 s at best with 300 ms. The demonstrations become slower and more hesitant, and so do the actions the policy learns. The most effective cure is a local loop that does not cross the network: the proprioception of a leader arm, a hand rendered locally in VR.

4第二个视角：UMI，不带机器人去采数据Second view: UMI, collecting data without a robot

主从臂要把整台机器人搬到采集现场。UMI（Universal Manipulation Interface，Chi 等，RSS 2024）换了个思路：做一个手持夹爪，人拿着它在任何地方做示范，机器人部署时在手腕上装一个同样的夹爪和同样的相机。具体做法：3D 打印的夹爪上装一台 GoPro（Hero 9）和 155° 的鱼眼镜头；视野边缘两面小镜子在同一张图里提供"隐式立体"视角；手指上的标记（fiducial markers）让夹爪开度可以从图像里连续读出；夹爪的 6 自由度位姿由视觉–惯性 SLAM 估计（基于 ORB-SLAM3，用 GoPro 内置的加速度计和陀螺仪）。夹爪物料约 73 美元，GoPro 及配件约 298 美元，整机 780 g。

没有机器人，也就没有"机器人的基座坐标系"。SLAM 给出的位姿在一个地图坐标系里，它的原点和朝向每次录制都可能不同，和部署时机器人站在哪里也没有关系。而腕部相机看到的世界，天然是在夹爪坐标系里的。UMI 的选择是让动作也在夹爪坐标系里：一段动作轨迹里的每个位姿，都相对于这段轨迹开始时的手爪位姿表示（"相对轨迹"），本体感觉也用相对量。这样，从观测到动作的映射与场景在世界里的位置、朝向无关。

下图在一张俯视的桌面上验证这件事。示范者从杯子前方走近、转向杯子、停在抓取位置（杯子在手指前方 8 cm）。我们用同一批 30 条示范（660 个（观测，动作）对）训练两个同样大小的 MLP：相对策略输入杯子在夹爪坐标系里的位置（腕部相机看到的），输出夹爪坐标系里的位移；绝对策略额外输入夹爪在世界坐标系里的位姿，输出世界坐标系里的位移。按"训练"让它们边学边看，再把桌子转 90° 或换一张桌子。

A leader–follower rig means carrying the whole robot to where the data is collected. UMI (Universal Manipulation Interface, Chi et al., RSS 2024) turns this around: build a hand-held gripper, let a person demonstrate with it anywhere, and at deployment mount the same gripper and the same camera on the robot's wrist. Concretely: a 3D-printed gripper carries a GoPro (Hero 9) with a 155° fisheye lens; two small mirrors at the edge of the view give "implicit stereo" views within the same image; fiducial markers on the fingers let the gripper width be read continuously from the image; and the gripper's 6-DoF pose comes from visual-inertial SLAM (based on ORB-SLAM3, using the GoPro's built-in accelerometer and gyroscope). The gripper's parts cost about $73, the GoPro and accessories about $298, and the device weighs 780 g.

Without a robot there is no "robot base frame". SLAM reports poses in a map frame whose origin and orientation may differ for every recording and have nothing to do with where the robot will stand at deployment. And the world a wrist camera sees is naturally expressed in the gripper frame. UMI's choice is to put the actions in the gripper frame too: every pose in an action trajectory is expressed relative to the gripper pose at the start of that trajectory (a "relative trajectory"), and the proprioception is relative as well. The map from observation to action then does not depend on where the scene sits in the world, or which way it faces.

The figure tests this on a top-down table. The demonstrator approaches the cup from the front, turns toward it and stops at the grasp pose (cup 8 cm in front of the fingers). The same 30 demonstrations (660 (observation, action) pairs) train two MLPs of the same size: the relative policy takes the cup's position in the gripper frame (what the wrist camera sees) and outputs a displacement in the gripper frame; the absolute policy additionally takes the gripper's pose in the world frame and outputs a displacement in the world frame. Press "Train" and watch them learn, then turn the table by 90° or move to another table.

FIG 3腕部相机 + 相对动作 vs 绝对动作Wrist camera + relative actions vs absolute actions

俯视图：拖动杯子、夹爪和它前方的朝向手柄TOP VIEW · drag the cup, the gripper and its heading handle

青色阴影 = 训练桌面上杯子出现的区域，青色细线 = 30 条示范轨迹；虚线方框 = 当前测试桌面上杯子出现的区域。琥珀色 = 相对策略的轨迹，紫色 = 绝对策略的轨迹，都从图中的夹爪出发、各走 30 步；绿色圆 = 杯子，青色扇形 = 腕部相机的视野。两个网络都是 [输入, 32, 32, 3]、tanh、Adam（学习率 0.004），批大小 32，训练到 1500 步自动停止。"评估"在当前桌面上随机摆 40 次杯子和起点，成功 = 杯子最终在手指前方 8 cm 处 2.5 cm 以内、偏角小于 15°。 Cyan shading = where cups appear on the training table, thin cyan lines = the 30 demonstrations; the dashed square = where cups appear on the current test table. Amber = the relative policy's path, purple = the absolute policy's path, both starting from the gripper shown and running 30 steps; the green disc is the cup and the cyan wedge the wrist camera's field of view. Both networks are [input, 32, 32, 3], tanh, Adam (learning rate 0.004), batch 32, and training stops by itself at 1500 steps. "Evaluate" draws 40 random cup and start placements on the current table; success = the cup ends within 2.5 cm of the point 8 cm in front of the fingers and within 15° of straight ahead.

试一试Try this

训练至少 1000 步，选"转 90° 的桌子"，按"评估"：相对策略成功率 ≥ 90%，绝对策略 ≤ 20%。在训练桌面上两者分别是多少？Train for at least 1000 steps, pick the "table turned 90°" and press "Evaluate": the relative policy succeeds at least 90% of the time and the absolute one at most 20%. What are the two rates on the training table?
训练至少 1000 步后，拖动夹爪和它的朝向手柄，让夹爪从一个与示范差别很大的方向接近杯子（起始朝向与 +x 方向相差超过 70°，即 $|\theta_0|>1.2$ rad；把夹爪放到杯子的另一侧，让杯子落在它前方的青色扇形里）：琥珀色轨迹成功，紫色轨迹失败。After at least 1000 training steps, drag the gripper and its heading handle so the gripper approaches the cup from a direction very different from the demonstrations (starting heading more than 70° from the +x direction, $|\theta_0|>1.2$ rad; put the gripper on another side of the cup, with the cup inside the cyan wedge in front of it): the amber path succeeds and the purple one fails.

相对动作解决了"在哪个坐标系"的问题，还有一个"在哪个时刻"的问题。在真实机器人上，策略看到的图像已经是几十毫秒前拍的（相机曝光、传输、解码），推理又要几十毫秒，指令发出后机械臂还要一段时间才执行到位。如果策略把图像当成"现在"、把它输出的动作序列当成"从现在开始立即生效"，那么对一个移动的目标，它永远慢半拍。UMI 的做法是延迟匹配：把各路观测都对齐到最慢的那一路（通常是相机）的时间戳；把动作按执行延迟提前发送，过时的动作直接丢弃。论文里动态抛掷任务的成功率因此从 57.5%（69/120）提高到 87.5%（105/120）。

下图模拟这件事：一个冰球在桌上来回滑动，策略以 10 Hz 运行，每次用最新的两帧图像（相隔 0.1 s）估计冰球的位置和速度，按匀速外推出一段动作轨迹，让夹爪停在冰球正上方。

Relative actions settle "in which frame"; there is also "at which moment". On a real robot the image a policy sees was taken tens of milliseconds ago (exposure, transfer, decoding), inference takes tens of milliseconds more, and after a command is sent the arm needs some time to get there. If the policy treats the image as "now" and its action sequence as "taking effect immediately", it is always half a step behind a moving target. UMI's answer is latency matching: align every observation stream to the timestamps of the slowest one (usually the camera), send actions ahead of time by the execution latency, and drop actions that are already out of date. In the paper this raised the success rate of a dynamic tossing task from 57.5% (69/120) to 87.5% (105/120).

The figure simulates it: a puck slides back and forth on a table, and a policy running at 10 Hz uses the newest two images (0.1 s apart) to estimate the puck's position and velocity, extrapolates at constant velocity into an action trajectory, and tries to keep the gripper right above the puck.

FIG 4延迟匹配：动作对齐到图像的时间戳Latency matching: timing actions from the image timestamp

俯视：冰球与两只夹爪TOP VIEW · the puck and two grippers

位置 x (m) vs 时间 (s)POSITION x (m) vs TIME (s)

青色 = 冰球的真实位置 $x(t)=0.3\sin(\omega t)$ m（滑块设定峰值速度 $0.3\,\omega$），紫色 = 不做匹配的夹爪（把图像当成现在、动作立即生效），琥珀色 = 做了延迟匹配的夹爪（动作从图像时间戳算起，并提前一个执行延迟发送）。下方的条形是延迟预算：红 = 相机，蓝 = 推理，绿 = 执行。误差是 $t\ge1$ s 之后夹爪与冰球距离的均方根，仿真步长 10 ms。 Cyan = the puck's true position $x(t)=0.3\sin(\omega t)$ m (the slider sets the peak speed $0.3\,\omega$), purple = the gripper without matching (image treated as now, actions taking effect at once), amber = the gripper with latency matching (actions timed from the image timestamp and sent one execution latency ahead). The bar underneath is the latency budget: red = camera, blue = inference, green = execution. The error is the root mean square distance between gripper and puck for $t\ge1$ s; the simulation steps at 10 ms.

试一试Try this

峰值速度保持在 0.25–0.30 m/s，把三段延迟加起来调到 300 ms 以上：不匹配的误差超过 5 cm，匹配后的误差小于 2 cm。延迟怎样分配在三段之间，结果有区别吗？With the peak speed at 0.25–0.30 m/s, raise the three latencies to a total of 300 ms or more: the unmatched error exceeds 5 cm while the matched one stays below 2 cm. Does it matter how the latency is split among the three stages?
找到连延迟匹配也救不了的情况：匹配后的误差超过 5 cm。需要调哪两个量？Find a case that even latency matching cannot rescue: the matched error exceeds 5 cm. Which two quantities do you need to raise?

换个角度看Another view

FIG 3 和 FIG 4 是同一个原则的两面：记录下的动作必须在部署时能以同样的意义执行。坐标系上，相对动作只依赖"夹爪此刻在哪"，这件事在采集和部署时都知道；绝对动作依赖一个部署时并不存在的地图坐标系。时间上，动作要和它所依据的那张图像对齐，而不是和"策略算完的时刻"对齐。UMI 还把采集和部署时的夹爪、相机、镜头做成完全一样，使观测本身也不变（论文里去掉鱼眼镜头后，杯子摆放任务从 20/20 降到 11/20）。

FIG 3 and FIG 4 are two sides of one principle: a recorded action must mean the same thing when it is executed at deployment. In space, a relative action depends only on "where the gripper is now", which is known both while collecting and while deploying; an absolute action depends on a map frame that does not exist at deployment. In time, an action must be aligned with the image it was based on, not with "the moment the policy finished". UMI also makes the gripper, camera and lens identical at collection and deployment, so the observation itself does not change (without the fisheye lens the paper's cup-arrangement task dropped from 20/20 to 11/20).

5第三个视角：数据的质量、多样性与数量Third view: data quality, diversity and size

采集预算是固定的：人时、场地、机器人。录 200 条示范，是在一张桌子上录 200 条，还是在 20 个厨房里各录 10 条？Lin 等（2024）用 UMI 采了 4 万多条示范、做了 1.5 万多次真机测试，结论很清楚：策略在新环境、新物体上的表现大致随训练环境数和物体数呈幂律增长；环境和物体的多样性远比示范的绝对数量重要；每个环境里的示范够了以后，再加几乎没有用。他们建议，对难度相近的任务，每个"环境–物体"组合录约 50 条，组合数达到 32 个时一般就够了；四个人用一个下午采的数据，就让两个任务在没见过的环境和物体上达到约 90% 的成功率。

下图是一个大约半分钟就能跑完的玩具版本。每个"环境"有自己的外观向量 $\mathbf z$（光照、桌布、背景），它会让相机看到的物体位置稍有偏移，也作为额外的输入特征；每个环境里物体集中出现在某一片区域。网络要从观测预测正确的动作（一个关于物体位置的光滑函数）。我们在 $K$ 个环境 × 每个 $N$ 条示范上训练，再在 40 个没见过的环境上测试。

A collection budget is fixed: person-hours, sites, robots. Should 200 demonstrations be 200 at one table, or 10 in each of 20 kitchens? Lin et al. (2024) collected over 40,000 demonstrations with UMI and ran over 15,000 real-robot rollouts, and the conclusion is clear: performance in new environments with new objects grows roughly as a power law in the number of training environments and objects; the diversity of environments and objects matters far more than the absolute number of demonstrations; and once each environment has enough demonstrations, more add little. For tasks of similar difficulty they recommend about 50 demonstrations per environment–object pair, with 32 pairs generally enough; data collected by four people in one afternoon brought two tasks to about 90% success in unseen environments with unseen objects.

The figure is a toy version that runs in about half a minute. Each "environment" has its own appearance vector $\mathbf z$ (lighting, tablecloth, background), which slightly shifts where the camera sees the object and is also an extra input feature; within each environment the objects tend to sit in one region. The network must predict the right action (a smooth function of the object's position) from the observation. We train on $K$ environments × $N$ demonstrations each and test on 40 environments it has never seen.

FIG 5多样性 vs 数量：在 $K\times N$ 网格上训练 108 个小网络Diversity vs quantity: 108 small networks on a $K\times N$ grid

新环境上的成功率（点击格子）SUCCESS IN NEW ENVIRONMENTS · click a cell

所选格子的训练数据与测试结果SELECTED CELL · training data and test results

左：横轴是环境数 $K$，纵轴是每个环境的示范数 $N$，格子越绿成功率越高；琥珀色框是所选格子，紫色斜线连起示范总数为 64 的格子。每格是 3 个随机种子的平均，每个种子一个 [5, 16, 16, 2] 的 tanh 网络、Adam（学习率 0.01）、批大小 32、600 步；测试集是 40 个新环境、各 10 个物体位置，预测动作与正确动作的距离小于 0.25 算成功。右：所选格子（第一个种子）的训练物体位置，按环境轮流着蓝、琥珀、紫、青色；小方块是测试位置，绿 = 成功，红 = 失败。"干净示范"的动作标签带标准差 0.05 的小噪声，"带噪示范"把这个独立噪声加大到标准差 0.3。采集时间 = $K\times$ 每个新环境的准备时间 + $K N\times$ 每条示范的时间（含复位）。按钮“类 ALOHA 平台”取每条 54 s（ALOHA 报告的 36–72 s 的中间值）和假设的每个新环境 30 min 搬运与布置；“类 UMI 手持”取每条 31 s（12 人时 / 1400 条，已包含在各地点之间移动的时间），不另加准备时间。 Left: environments $K$ across, demonstrations per environment $N$ up; greener cells succeed more often; the amber frame is the selected cell and the purple diagonal joins the cells with 64 demonstrations in total. Each cell averages 3 random seeds, each a [5, 16, 16, 2] tanh network trained with Adam (learning rate 0.01), batch 32, for 600 steps; the test set is 40 new environments with 10 object positions each, and a prediction within 0.25 of the right action counts as a success. Right: the training object positions of the selected cell (first seed), coloured by environment in turn blue, amber, purple, cyan; the small squares are test positions, green = success, red = failure. The action labels of "clean demos" carry small independent noise (standard deviation 0.05); "noisy demos" raises it to 0.3. Collection time = $K\times$ setup time per new environment + $KN\times$ time per demonstration (resets included). The "ALOHA-like rig" button uses 54 s per demonstration (the middle of ALOHA's reported 36–72 s) and an assumed 30 min to move and set up the rig per new environment; "UMI-like hand-held" uses 31 s per demonstration (12 person-hours / 1400 demonstrations, travel between locations already included) and no extra setup.

试一试Try this

按"运行"，等网格算完（"干净示范"）。在示范总数正好为 64 的五个格子（紫色斜线）里，点出成功率最高的那个。它比 $K=2,\,N=32$ 高多少？Press "Run" and let the grid finish ("clean demos"). Among the five cells with exactly 64 demonstrations (the purple diagonal), click the one with the highest success. How much better is it than $K=2,\,N=32$?
切到"带噪示范"，等网格重新算完，在 $K=32$ 一列里点一个成功率 ≥ 80% 的格子。和干净示范相比，这一列需要的 $N$ 变了吗？为什么？Switch to "noisy demos", let the grid recompute, and click a cell in the $K=32$ column with success of at least 80%. Compared with clean demos, has the $N$ this column needs changed? Why?

核心洞见Key insight

干净示范时，固定 64 条总数，$K=2,N=32$ 在新环境上只成功 26%，$K=32,N=2$ 成功 93%；$K=16$ 时，$N$ 从 1 加到 8，成功率从 39% 升到 97%，再往上就几乎不动了。原因是 16.2 节讲的泛化：环境太少时，外观特征 $\mathbf z$ 和物体所在的区域一一对应，网络可以靠"认出是哪个环境"来拟合训练数据，到了新环境就外推失败；环境多了，$\mathbf z$ 与动作无关这一事实才在数据里显现出来。示范带噪时，数量重新变得重要：$K=32$ 时 $N=1$ 只有 18%，$N=32$ 才到 92%，因为独立的噪声要靠多次平均才能消掉。

With clean demonstrations and 64 in total, $K=2,N=32$ succeeds only 26% of the time in new environments while $K=32,N=2$ succeeds 93%; at $K=16$, raising $N$ from 1 to 8 lifts success from 39% to 97%, and beyond that it barely moves. The reason is the generalization of Lesson 16.2: with few environments the appearance features $\mathbf z$ correspond one-to-one with where the objects sit, so the network can fit the training data by "recognizing which environment this is" and then extrapolates badly in a new one; only with many environments does the data reveal that $\mathbf z$ is irrelevant to the action. With noisy demonstrations quantity matters again: at $K=32$, $N=1$ gives 18% and $N=32$ is needed for 92%, because independent noise only cancels by averaging over many samples.

质量同样要紧，而且常常比数量更难补救。好的示范一致（同一个任务用同一种策略，不同操作员之间也尽量一致；不一致的示范会变成多模态，MSE 会把它们平均掉，见 22.1 节的 FIG 5，生成式策略见 22.2 节）、流畅（没有长时间的停顿；停顿的帧教会策略"原地不动"）、可复现（只用机器人部署时也能得到的信息完成，例如操作员不要靠侧面才看得见的视角）、并且最好包含一些从偏差中恢复的片段（这正是 DAgger 想要的数据，见 22.1 节）。

Quality matters as much, and is often harder to fix after the fact than quantity. Good demonstrations are consistent (one task, one strategy, and as similar as possible across operators; inconsistent demonstrations become multimodal and MSE averages them, see FIG 5 of Lesson 22.1 and the generative policies of Lesson 22.2), smooth (no long pauses; paused frames teach the policy to stand still), reproducible (completed using only information the robot will have at deployment, so the operator should not rely on a side view the robot lacks), and ideally include some recoveries from deviations (exactly the data DAgger asks for, see Lesson 22.1).

6再看数学The math

关节空间映射与任务空间重定向Joint-space mapping and task-space retargeting

关节空间映射只需逐个关节标定一个符号和零位偏移（$D$ 是对角的 $\pm1$ 矩阵，$\mathbf b$ 是偏移）：

Joint-space mapping only needs a sign and a zero offset calibrated per joint ($D$ is a diagonal matrix of $\pm1$, $\mathbf b$ the offsets):

$$ \hl{jc}{\mathbf q_f = D\,\mathbf q_\ell+\mathbf b}. $$

如果从臂的每一根连杆都是主臂的 $1/s$ 倍（关节轴的排列相同），那么正运动学满足 $\mathbf p_f(\mathbf q)=\mathbf p_\ell(\mathbf q)/s$、$R_f(\mathbf q)=R_\ell(\mathbf q)$：同样的关节角给出同样朝向、按比例放大的手爪位置。比例不同时这个等式不再成立，误差随姿态变化（FIG 1 里伸直时只剩总长度不同造成的 4 cm，弯曲时可以超过 10 cm）。

If every link of the follower is $1/s$ times the leader's (with the same arrangement of joint axes), forward kinematics satisfies $\mathbf p_f(\mathbf q)=\mathbf p_\ell(\mathbf q)/s$ and $R_f(\mathbf q)=R_\ell(\mathbf q)$: the same joint angles give the same orientation and a proportionally enlarged tool position. With different proportions the identity no longer holds and the error varies with posture (in FIG 1 only the 4 cm from the different total length remains when straight, while bent it can exceed 10 cm).

任务空间重定向先构造从臂手爪的目标位姿。VR 手柄、手机这类接口一般带"离合"：按下按钮的时刻 $t_0$ 记下人手位姿 $H_{t_0}$ 和从臂手爪位姿 $T_{f,t_0}$，之后只传递人手的相对运动（位移部分乘缩放系数）：

Task-space retargeting first builds a target pose for the follower's tool. Interfaces such as VR controllers and phones usually have a "clutch": at the moment $t_0$ the button is pressed, record the hand pose $H_{t_0}$ and the follower's tool pose $T_{f,t_0}$, and from then on pass on only the hand's relative motion (with its translation multiplied by a scale factor):

$$ \hl{ik}{T^\star_t} \;=\; T_{f,t_0}\;\mathcal S_{1/s}\!\big(H_{t_0}^{-1}H_t\big),\qquad \Delta\mathbf q=J^\top\big(JJ^\top+\lambda^2 I\big)^{-1}\mathbf e\big(\mathbf q,T^\star_t\big), $$

其中 $\mathcal S_{1/s}$ 只把位移部分除以 $s$，$\mathbf e$ 是 7.5 节的位姿误差，右式是那里的阻尼最小二乘一步。每一帧从上一帧的解热启动迭代几次。阻尼 $\lambda$ 让奇异位形附近的步长有界，代价是目标够不到时，从臂只会"尽量靠近"，停在残差不为零的地方（FIG 1 的第三个任务）。离合还解决了另一个问题：人手的工作空间和从臂的不同，松开离合、把手挪回舒服的位置、再按下，就能"重新对齐"。

Here $\mathcal S_{1/s}$ divides only the translation by $s$, $\mathbf e$ is the pose error of Lesson 7.5, and the right-hand formula is one damped least-squares step from there. Every frame warm-starts from the previous solution and iterates a few times. The damping $\lambda$ keeps steps bounded near singular postures, at the price that when the target is out of reach the follower only gets "as close as it can" and stops with a non-zero residual (the third task of FIG 1). The clutch also solves another problem: the hand's workspace differs from the follower's, so you release the clutch, move the hand back to a comfortable place and press again to "re-align".

延迟给增益设的上限The ceiling latency puts on the gain

把 FIG 2 的操作员模型线性化：手速正比于感知误差，从臂位置是手速的积分，回路总延迟 $\tau=\tau_h+\tau_s$（人的反应 + 系统延迟），开环传递函数是

Linearize FIG 2's operator model: hand speed is proportional to the perceived error, the follower's position integrates the hand speed, and the loop's total delay is $\tau=\tau_h+\tau_s$ (human reaction + system latency). The open-loop transfer function is

$$ L(s)=\frac{K\,e^{-s\hl{loop}{\tau}}}{s},\qquad |L(j\omega)|=\frac{K}{\omega}\;\Rightarrow\;\omega_c=K,\qquad \angle L(j\omega_c)=-\frac{\pi}{2}-K\tau . $$

相位裕度 $\pi/2-K\tau$ 必须为正，所以 $K<\pi/(2\tau)$。闭环的时间常数大约是 $1/K$，于是最快的完成时间至少和 $\tau$ 成正比：延迟翻倍，能用的增益减半，完成时间大约翻倍。FIG 2 的数字（再加上 30 ms 的从臂伺服和 0.6 m/s 的手速上限）与这个估计一致：系统延迟 100 ms 时 $\tau=0.25$ s，上限 $K<6.3$，$K=6$ 时已经停不下来。"自己的手"闭环时，从视频进入回路的延迟消失，$\tau$ 只剩 $\tau_h$。这也是双边遥操作（带力反馈）理论的出发点：Anderson 与 Spong（1989）证明了有延迟时朴素的力反馈会失稳，并用基于无源性的散射理论方法恢复稳定（后来 Niemeyer 与 Slotine 把它改写成"波变量"的形式）。

The phase margin $\pi/2-K\tau$ must be positive, so $K<\pi/(2\tau)$. The closed loop's time constant is about $1/K$, so the fastest completion time is at least proportional to $\tau$: double the delay, halve the usable gain, roughly double the completion time. FIG 2's numbers (which add a 30 ms follower servo and a 0.6 m/s hand-speed limit) agree with this estimate: at 100 ms of system latency $\tau=0.25$ s, the ceiling is $K<6.3$, and at $K=6$ the gripper already fails to settle. With "own hand" feedback the delay entering through the video disappears and $\tau$ is just $\tau_h$. This is also the starting point of bilateral (force-feedback) teleoperation theory: Anderson and Spong (1989) showed that naive force feedback goes unstable under delay and restored stability with a passivity-based method from scattering theory (later recast as "wave variables" by Niemeyer and Slotine).

相对动作为什么不依赖坐标系Why relative actions do not depend on the frame

设 $T_t\in\SE$ 是夹爪在某个世界坐标系（SLAM 地图或机器人基座）里的位姿（5.3 节）。一段从 $t_0$ 开始的动作轨迹，用相对位姿表示为

Let $T_t\in\SE$ be the gripper's pose in some world frame (the SLAM map or the robot base; Lesson 5.3). An action trajectory starting at $t_0$, written in relative poses, is

$$ \hl{rel}{\Delta_k=T_{t_0}^{-1}\,T_{t_0+k}},\qquad k=1,\dots,H . $$

换一个世界坐标系，相当于所有位姿左乘同一个刚体变换 $G$：$T_t\mapsto GT_t$。相对位姿不变：

Changing the world frame left-multiplies every pose by the same rigid transform $G$: $T_t\mapsto GT_t$. The relative poses do not change:

$$ (GT_{t_0})^{-1}(GT_{t_0+k})=T_{t_0}^{-1}G^{-1}G\,T_{t_0+k}=\Delta_k ,\qquad \hl{abs}{T_{t_0+k}\mapsto GT_{t_0+k}} . $$

腕部相机看到的物体位置 $\mathbf o^{\,c}=T_{t}^{-1}\mathbf o$ 同样不变。于是"观测 → 相对动作"这一对数据在整个场景平移、旋转之后保持原样，策略在任何地方看到的都是训练分布里的输入；而绝对动作 $T_{t_0+k}$ 跟着 $G$ 变了，一个在训练桌面上学的网络在转过的桌面上就要外推（FIG 3）。部署时，机器人以自己此刻的手爪位姿 $T^{r}_{t_0}$ 为起点执行 $T^{\star}_{t_0+k}=T^{r}_{t_0}\,\Delta_k$，只需要它自己的正运动学。UMI 用"相对于这段轨迹起点"的写法，而不是逐步的增量 $\delta_k=T_{t_0+k-1}^{-1}T_{t_0+k}$：增量要连乘 $T_{t_0+k}=T_{t_0}\delta_1\cdots\delta_k$，每一步的误差都会累积。论文里杯子摆放任务用增量动作时从 20/20 降到 16/20。

The object position seen by the wrist camera, $\mathbf o^{\,c}=T_{t}^{-1}\mathbf o$, is unchanged too. So an "observation → relative action" pair stays the same after the whole scene is shifted and rotated, and the policy sees inputs from its training distribution wherever it is; the absolute action $T_{t_0+k}$, by contrast, changes with $G$, and a network trained on one table has to extrapolate on a turned one (FIG 3). At deployment the robot starts from its own current tool pose $T^{r}_{t_0}$ and executes $T^{\star}_{t_0+k}=T^{r}_{t_0}\,\Delta_k$, which needs only its own forward kinematics. UMI writes actions relative to the start of the trajectory rather than as step-by-step deltas $\delta_k=T_{t_0+k-1}^{-1}T_{t_0+k}$: deltas must be chained, $T_{t_0+k}=T_{t_0}\delta_1\cdots\delta_k$, and every step's error accumulates. In the paper the cup-arrangement task dropped from 20/20 to 16/20 with delta actions.

延迟匹配Latency matching

设图像在 $t_c$ 时刻拍下，从拍摄到策略拿到它是 $\tau_o$，推理 $\tau_i$，指令到机械臂执行到位 $\tau_e$。策略预测的是"从图像时刻起"的轨迹 $\hat x(t_c+u)$。不做匹配时，它在 $t_c+\tau_o+\tau_i$ 时刻之后 $u$ 秒才发出 $\hat x(t_c+u)$，再晚 $\tau_e$ 执行，于是夹爪位置 $x_g$ 满足下式左边；做匹配时，在 $t$ 时刻发送的指令 $u(t)$ 是右式：

Let an image be taken at time $t_c$; it reaches the policy $\tau_o$ later, inference takes $\tau_i$, and a command takes $\tau_e$ to be executed by the arm. The policy predicts a trajectory "from the image moment", $\hat x(t_c+u)$. Without matching it issues $\hat x(t_c+u)$ at $u$ seconds after $t_c+\tau_o+\tau_i$, executed a further $\tau_e$ later, so the gripper position $x_g$ obeys the left formula; with matching, the command $u(t)$ sent at time $t$ is the right one:

$$ x_g(t)=\hat x\big(t-(\tau_o+\tau_i+\tau_e)\big)\;\Rightarrow\; |x_g(t)-x(t)|\approx |\dot x|\,(\tau_o+\tau_i+\tau_e),\qquad \hl{lm}{u(t)=\hat x\big(t+\tau_e\big)} . $$

延迟匹配在 $t$ 时刻发送"为 $t+\tau_e$ 时刻准备的那一步"，并丢弃目标时刻已经过去的步，于是 $x_g(t)=\hat x(t)$：剩下的误差只来自预测本身（这里是匀速外推遇到加速度，大约 $\tfrac12|\ddot x|(\tau_o+\tau_i+\tau_e)^2$ 再加上用两帧估计速度的误差）。在 FIG 4 默认的 200 ms、0.3 m/s 下，不匹配的误差 3.9 cm，匹配后 1.0 cm；三段延迟只以总和出现，所以怎么分配结果都一样。

Latency matching sends, at time $t$, "the step meant for time $t+\tau_e$" and drops steps whose target time has passed, so $x_g(t)=\hat x(t)$: the remaining error comes only from the prediction itself (here constant-velocity extrapolation meeting acceleration, about $\tfrac12|\ddot x|(\tau_o+\tau_i+\tau_e)^2$, plus the error of estimating velocity from two frames). At FIG 4's default 200 ms and 0.3 m/s the unmatched error is 3.9 cm and the matched one 1.0 cm; the three latencies only appear as a sum, so the split makes no difference.

多样性与数量的一个简单模型A simple model of diversity and quantity

把新环境上的误差粗略拆成两项：一项来自"环境之间的差异没被覆盖"，随环境数 $K$ 下降；一项来自标签噪声，随总样本数下降：

Roughly split the error in new environments into two terms: one from "differences between environments not covered", which falls with the number of environments $K$, and one from label noise, which falls with the total number of samples:

$$ \mathcal E(K,N)\;\approx\;c\,\hl{div}{K}^{-\alpha}\;+\;\frac{\sigma^2}{K N}. $$

示范干净（$\sigma$ 小）时第一项主导，$N$ 几乎不起作用，增加 $K$ 才有用；示范带噪时第二项变大，$N$ 重新变得重要（FIG 5 的两种设置）。这只是帮助思考的示意，不是 Lin 等论文里拟合出的公式；他们报告的是成功率随环境数、物体数近似呈幂律关系。21.2 节会更系统地讨论规模定律。

With clean demonstrations (small $\sigma$) the first term dominates, $N$ barely matters and only more $K$ helps; with noisy demonstrations the second term grows and $N$ matters again (FIG 5's two settings). This is a sketch to think with, not the formula fitted in Lin et al.'s paper; they report that performance follows an approximate power law in the numbers of environments and objects. Lesson 21.2 treats scaling laws more systematically.

7机器人中的它：采集系统与数据格式In robotics: collection systems and data formats

ALOHA 家族。ALOHA（Zhao 等，RSS 2023）：两对主从臂，四台 Logitech C922x 网络摄像头（两台在从臂手腕上，一台在前、一台在顶），480×640 图像，遥操作与记录都在 50 Hz；每个任务 50 条示范，配合 ACT（22.2 节）用约 10 分钟的示范数据在六个真实任务上达到 80–90% 的成功率。ALOHA 2（Google DeepMind 等，2024）改进了硬件：主臂夹爪改成低摩擦直线导轨，打开力从 14.68 N 降到 0.84 N；从臂夹爪指尖夹持力从 12.8 N 升到 27.9 N；用被动的伸缩器替代橡皮筋做重力补偿；相机换成四台 Intel RealSense D405；并开放了带系统辨识的 MuJoCo 模型。Mobile ALOHA（Fu、Zhao、Finn，CoRL 2024）把 ALOHA 装上 AgileX Tracer 移动底盘，操作员的腰部与底盘相连，边用双手操纵主臂边用身体推着底盘走（"全身遥操作"），整机预算 3.2 万美元以内；与 825 条静态 ALOHA 数据联合训练后，用 ACT 时"叫电梯"从 0% 提高到 95%，"冲洗平底锅"从 0% 提高到 80%。

其他接口。RoboTurk（Mandlekar 等，CoRL 2018）用 iPhone 一类的手机做 6 自由度遥操作，让远程众包工人采了 137.5 小时的数据，其中 2200 多条成功示范。GELLO（Wu 等，2023）的 12 人用户研究里，GELLO、VR 手柄和 3D 鼠标的平均任务成功率分别是 0.92、0.72、0.63。DROID（Khazatsky 等，2024）由 50 名采集员在 12 个月里于北美、亚洲、欧洲采了 7.6 万条轨迹、350 小时、564 个场景、86 个任务。AirExo（Fang 等，2024）是与机械臂运动学匹配的低成本外骨骼，和主从臂同属"关节复制"一类。Open-TeleVision（Cheng 等，CoRL 2024）在 Unitree H1 和 Fourier GR-1 人形机器人上把立体相机画面实时送进头显，并让机器人跟随操作员手臂和手的运动。LeRobot 的 SO-100 / SO-101 低成本主从臂套件，用 lerobot-record 一条命令同时遥操作和录制数据集。

数据格式。两种格式最常见，它们存的东西大同小异：一条条轨迹（episode），每条是一串带时间戳的步（step）。

The ALOHA family. ALOHA (Zhao et al., RSS 2023): two leader–follower pairs and four Logitech C922x webcams (two on the follower wrists, one in front, one on top) at 480×640, with teleoperation and recording both at 50 Hz; 50 demonstrations per task, and with ACT (Lesson 22.2) about 10 minutes of demonstrations reach 80–90% success on six real tasks. ALOHA 2 (Google DeepMind and others, 2024) improved the hardware: the leader gripper moved to a low-friction linear rail, cutting the opening force from 14.68 N to 0.84 N; the follower gripper's fingertip force rose from 12.8 N to 27.9 N; passive retractors replaced the rubber bands for gravity compensation; the cameras became four Intel RealSense D405; and a system-identified MuJoCo model was released. Mobile ALOHA (Fu, Zhao, Finn, CoRL 2024) puts ALOHA on an AgileX Tracer mobile base; the operator's waist is tethered to the base, so they drive the leaders with both hands while pushing the base with their body ("whole-body teleoperation"), all within a $32k budget; co-training with 825 static ALOHA demonstrations raised "Call Elevator" from 0% to 95% and "Rinse Pan" from 0% to 80% with ACT.

Other interfaces. RoboTurk (Mandlekar et al., CoRL 2018) uses phones such as the iPhone for 6-DoF teleoperation and let remote crowd workers collect 137.5 hours of data, including over 2,200 successful demonstrations. In GELLO's 12-person user study (Wu et al., 2023) the average task success was 0.92 for GELLO, 0.72 for VR controllers and 0.63 for a 3D mouse. DROID (Khazatsky et al., 2024) had 50 collectors gather 76k trajectories, 350 hours, 564 scenes and 86 tasks over 12 months across North America, Asia and Europe. AirExo (Fang et al., 2024) is a low-cost exoskeleton kinematically matched to the arm, in the same "copy joints" family as leader arms. Open-TeleVision (Cheng et al., CoRL 2024) streams stereo camera images to a headset in real time on Unitree H1 and Fourier GR-1 humanoids, with the robot following the operator's arm and hand motion. LeRobot's low-cost SO-100 / SO-101 leader–follower kits teleoperate and record a dataset with a single lerobot-record command.

Data formats. Two formats are the most common, and they store much the same thing: episodes, each a sequence of timestamped steps.

	RLDS	LeRobotDataset (v3.0)
来源Origin	Ramos 等（Google，2021），基于 TFDS / `tf.data`Ramos et al. (Google, 2021), built on TFDS / `tf.data`	Hugging Face LeRobot，PyTorch，托管在 Hugging Face HubHugging Face LeRobot, PyTorch, hosted on the Hugging Face Hub
结构Structure	episode = 元数据 + 一串 step；每个 step 有 `observation`、`action`、`reward`、`discount`、`is_first`、`is_last`、`is_terminal`，其中只有 `is_first` 和 `is_last` 是必需的episode = metadata + a sequence of steps; each step has `observation`, `action`, `reward`, `discount`, `is_first`, `is_last` and `is_terminal`, of which only `is_first` and `is_last` are mandatory	低维信号（状态、动作、时间戳）存在 Parquet 里，相机画面编码成 MP4；`meta/` 下有 schema 与帧率（`info.json`）、归一化统计（`stats.json`）、任务描述和每条 episode 的起止偏移low-dimensional signals (states, actions, timestamps) in Parquet, camera frames encoded as MP4; `meta/` holds the schema and frame rate (`info.json`), normalization stats (`stats.json`), task descriptions and each episode's offsets
典型用途Typical use	Open X-Embodiment 用它统一了 60 个数据集Open X-Embodiment used it to unify 60 datasets	用 `lerobot-record` 录制，`delta_timestamps` 按秒取时间窗口（历史帧、动作块）recorded with `lerobot-record`; `delta_timestamps` fetches time windows in seconds (history frames, action chunks)

# LeRobot：按时间（秒）取历史观测与未来动作块（22.1 节的动作分块）
ds = LeRobotDataset(repo_id, delta_timestamps={
    "observation.images.wrist": [-0.1, 0.0],                 # 两帧历史
    "action": [k / fps for k in range(16)],                   # 16 步动作块
})# LeRobot: fetch past observations and a future action chunk by time in seconds (action chunking, Lesson 22.1)
ds = LeRobotDataset(repo_id, delta_timestamps={
    "observation.images.wrist": [-0.1, 0.0],                 # two frames of history
    "action": [k / fps for k in range(16)],                   # a 16-step action chunk
})

真实系统里In real systems

Open X-Embodiment（Open X-Embodiment Collaboration，2023）把 34 个实验室的 60 个已有数据集统一成 RLDS 格式：超过 100 万条真实机器人轨迹，22 种机器人形态，21 家机构，527 种技能（160,266 个任务）。在它上面训练的 RT-1-X 在小数据领域比只用单个数据集训练的 RT-1 或原方法好 50%，RT-2-X 在涌现技能评测上是 RT-2 的 3 倍。跨形态混合数据时，动作空间（关节 / 末端、绝对 / 相对、控制频率）和坐标系约定各不相同，统一它们是工作量的大头，这正是 FIG 3 和 FIG 4 讨论的问题在数据集层面的样子。工程上：给每一路数据记录采集时刻的时间戳（而不是收到的时刻）；把相机内外参、控制频率、动作的坐标系和单位写进元数据；录制时就标记失败和被打断的 episode（RLDS 为此留了 invalid 之类的元数据字段）。

Open X-Embodiment (Open X-Embodiment Collaboration, 2023) unified 60 existing datasets from 34 labs into RLDS: over 1M real robot trajectories, 22 robot embodiments, 21 institutions and 527 skills (160,266 tasks). RT-1-X trained on it beats RT-1 or the original methods trained on single datasets by 50% in the small-data domain, and RT-2-X does 3× better than RT-2 on emergent-skill evaluations. When data from many embodiments is mixed, action spaces (joint / end-effector, absolute / relative, control rate) and frame conventions all differ, and unifying them is most of the work: the questions of FIG 3 and FIG 4 at the scale of a dataset. In practice: timestamp every stream at capture time (not arrival time); write camera intrinsics and extrinsics, control rate, and the action frame and units into the metadata; and flag failed or interrupted episodes at recording time (RLDS reserves metadata fields such as invalid for this).

8常见误区Common pitfalls

误区 1：把遥操作数据当成"最优示范"Pitfall 1: treating teleoperation data as "optimal demonstrations"

示范里有停顿、犹豫、试探和纠正。行为克隆会忠实地学走这些：长时间静止的帧让策略在某些状态下"不动"，不同操作员的不同策略让动作分布变成多模态。FIG 2 里延迟 300 ms 时，默认增益的示范在目标附近来回摆了 6 s 都没停下，这样的轨迹进了数据集，策略也会摆。录制前统一操作规范，录制后裁掉首尾静止段、剔除失败和被打断的 episode，比事后加数据更有效。

Demonstrations contain pauses, hesitations, probing and corrections, and behaviour cloning faithfully learns them: long stretches of motionless frames make the policy "freeze" in certain states, and different operators' different strategies make the action distribution multimodal. In FIG 2 at 300 ms of latency, the default-gain demonstration swings around the target for the full 6 s without settling; put trajectories like that into a dataset and the policy will swing too. Agreeing on an operating protocol before recording, and trimming idle heads and tails and removing failed or interrupted episodes afterwards, does more than adding data later.

误区 2：用"收到的时间"给数据打时间戳Pitfall 2: timestamping data at arrival

相机、关节编码器、夹爪传感器的延迟各不相同，按"程序收到的时刻"对齐，图像和关节状态之间就差了几十毫秒。手爪移动时，图里物体相对手爪的位置和本体感觉对不上，训练时学到的就是一个被模糊的映射；部署时延迟又不一样，误差就系统化了（FIG 4 里匹配与不匹配的误差差了约 4 倍）。记录采集时刻，按最慢的流对齐，并在部署时测一遍实际延迟。

Cameras, joint encoders and gripper sensors all have different latencies; aligning them by "when the program received them" leaves tens of milliseconds between an image and the joint state. While the gripper moves, the object's position relative to the gripper in the image no longer agrees with the proprioception, and training learns a blurred map; at deployment the latencies differ again and the error becomes systematic (in FIG 4 the matched and unmatched errors differ by about 4×). Record capture times, align to the slowest stream, and measure the actual latencies at deployment.

误区 3：在部署时不存在的坐标系里记录动作Pitfall 3: recording actions in a frame that does not exist at deployment

SLAM 地图坐标系、某次标定的世界坐标系、采集时机器人基座的位置，部署时都不一定还在。FIG 3 里在训练桌面上 100% 成功的绝对策略，桌子转 90° 后成功率是 0%，相对策略仍然 100%。同样要小心的是相机安装：采集时腕部相机的位置、镜头、视场与部署时稍有不同，观测就离开了训练分布。

The SLAM map frame, the world frame of one particular calibration, the robot base position during collection: none of them is guaranteed to exist at deployment. In FIG 3 the absolute policy that succeeds 100% of the time on the training table succeeds 0% of the time once the table turns 90°, while the relative policy stays at 100%. Camera mounting deserves the same care: if the wrist camera's placement, lens or field of view differs slightly between collection and deployment, the observations leave the training distribution.

误区 4：在一个场景里录完所有数据Pitfall 4: recording everything in one scene

在实验室同一张桌子上录 200 条，测试时成功率很高，换个房间就崩溃。网络学到的是"这个场景里的这个任务"，背景、光照、桌布都成了它的线索。FIG 5 里 $K=1$ 时，$N$ 从 1 加到 32，新环境上的成功率只从 3% 到 19%。评估时一定要用没见过的环境和物体；采集时把预算花在多样性上，每个环境录"够用"即可。

Record 200 demonstrations at the same lab table and the test success looks great, until you move to another room. The network has learned "this task in this scene", with background, lighting and tablecloth all serving as cues. In FIG 5 at $K=1$, raising $N$ from 1 to 32 moves success in new environments only from 3% to 19%. Always evaluate in unseen environments with unseen objects; when collecting, spend the budget on diversity and record "enough" per environment.

误区 5：示范用了机器人做不到或看不到的东西Pitfall 5: demonstrations that use what the robot cannot do or see

手持夹爪很自由，人可以做出机械臂够不到、关节转不过去、或速度和加速度超出电机能力的动作；操作员可以从侧面、从高处看清楚，而机器人只有腕部和固定相机的画面；操作员的手能感觉到接触力，机器人可能没有力传感器。这些示范在训练集里看着没问题，策略却无法复现。采集后要检查动作的运动学可行性（例如对每条轨迹解一遍 IK），并让操作员尽量只依赖机器人自己能获得的观测。

A hand-held gripper is free: a person can make motions the arm cannot reach, joints cannot turn through, or motors cannot match in speed and acceleration; operators can see clearly from the side or from above, while the robot has only its wrist and fixed cameras; an operator's hand feels contact forces the robot may have no sensor for. Such demonstrations look fine in the training set, but the policy cannot reproduce them. After collection check the actions for kinematic feasibility (for example, solve IK along every trajectory), and have operators rely as much as possible on the observations the robot itself will have.

9小练习Check yourself

1. ALOHA 的从臂直接复制主臂的关节角，而不是对主臂手爪位姿解逆运动学。主要原因是什么？1. ALOHA's followers copy the leaders' joint angles instead of solving inverse kinematics for the leaders' tool poses. What is the main reason?

选项 C。ALOHA 论文明确提到现成 IK 在奇异位形附近经常失败。关节复制的前提恰恰是两条臂结构相似（所以选项 D 反了）；结构不同时，只能用任务空间重定向（FIG 1）。带宽和速度都不是原因。Option C. The ALOHA paper says explicitly that off-the-shelf IK often fails near singularities. Copying joints requires the two arms to be similar in structure (so option D is backwards); for different structures you need task-space retargeting (FIG 1). Neither bandwidth nor speed is the reason.

2. 把操作员看成增益为 $K$ 的比例控制器，从臂是一个积分器。回路总延迟从 0.25 s 增加到 0.5 s，能保持稳定的最大增益怎样变化？2. Treat the operator as a proportional controller with gain $K$ and the follower as an integrator. If the total loop delay grows from 0.25 s to 0.5 s, how does the largest stable gain change?

选项 B。$L(s)=Ke^{-s\tau}/s$ 的穿越频率是 $\omega_c=K$，相位裕度 $\pi/2-K\tau>0$ 给出 $K<\pi/(2\tau)$：$\pi/0.5\approx6.3$，$\pi/1.0\approx3.1$。延迟确实只改相位，但正是因为它吃掉了 $K\tau$ 的相位，增益上限才和 $\tau$ 成反比（所以选项 A 错）。Option B. $L(s)=Ke^{-s\tau}/s$ crosses over at $\omega_c=K$, and the phase margin $\pi/2-K\tau>0$ gives $K<\pi/(2\tau)$: $\pi/0.5\approx6.3$, $\pi/1.0\approx3.1$. Delay does change only the phase, but because it eats $K\tau$ of phase, the gain ceiling is inversely proportional to $\tau$ (so option A is wrong).

3. 用 UMI 采集的数据，部署时机器人基座放在和示范者完全不同的位置和朝向。为什么"相对于轨迹起点"的动作仍然可以直接执行？3. Data was collected with UMI; at deployment the robot base stands at a completely different position and orientation from the demonstrator. Why can actions "relative to the start of the trajectory" still be executed directly?

选项 A。相对位姿对世界坐标系的选择不变（6 节），腕部相机的观测也不变，所以观测–动作对在新的位置和朝向下与训练时相同。SLAM 地图与基座没有固定关系（选项 B 错）；部署时也不需要 SLAM（选项 D 错）。数值大小不是关键（选项 C）。Option A. Relative poses do not depend on the choice of world frame (Section 6), nor do the wrist camera's observations, so observation–action pairs at the new position and orientation are the same as in training. The SLAM map has no fixed relation to the base (option B is wrong), and deployment needs no SLAM (option D is wrong). Magnitude is not the point (option C).

4. 一个物体以 0.5 m/s 匀速移动，相机、推理和执行延迟合计 0.2 s。策略把图像当成"现在"、动作立即生效（不做延迟匹配）。夹爪大约落后物体多少？4. An object moves at a constant 0.5 m/s, and camera, inference and execution latency total 0.2 s. The policy treats the image as "now" and its actions as taking effect immediately (no latency matching). Roughly how far behind the object is the gripper?

选项 D。滞后 ≈ 速度 × 总延迟 = 0.5 × 0.2 = 0.1 m。三段延迟以总和出现。物体匀速时，延迟匹配加上两帧估计的速度可以几乎完全消除这个误差；FIG 4 里冰球有加速度，所以匹配后仍留下外推误差。Option D. The lag ≈ speed × total latency = 0.5 × 0.2 = 0.1 m. The three latencies appear as a sum. For constant velocity, latency matching with a two-frame velocity estimate removes this error almost entirely; FIG 4's puck accelerates, so an extrapolation error remains after matching.

5. 你有采 200 条示范的预算，目标是在没见过的厨房里抓没见过的杯子。示范都很干净。依据 Lin 等（2024）和 FIG 5，哪种分配最可能泛化得最好？5. You can afford 200 demonstrations and want the policy to grasp unseen cups in unseen kitchens. The demonstrations are clean. Based on Lin et al. (2024) and FIG 5, which allocation is most likely to generalize best?

选项 D。对新环境、新物体的泛化主要随环境和物体的数量增长，每个环境里的示范一旦够用，再多收益很小。FIG 5 里固定 64 条时，$K=2,N=32$ 只有 26%，$K=32,N=2$ 达到 93%。只有示范很嘈杂或任务很难、每个环境需要更多样本才能学会时，才应该在每个环境里多录。Option D. Generalization to new environments and objects grows mainly with the number of environments and objects; once each environment has enough demonstrations, more help little. In FIG 5 with 64 in total, $K=2,N=32$ reaches only 26% while $K=32,N=2$ reaches 93%. Only when demonstrations are noisy, or the task is hard enough that each environment needs more samples to learn, should you record more per environment.

10延伸阅读Further reading

T. Z. Zhao, V. Kumar, S. Levine & C. Finn, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, RSS 2023 — ALOHA 硬件与 ACTThe ALOHA hardware and ACT
Z. Fu, T. Z. Zhao & C. Finn, Mobile ALOHA: Learning Bimanual Mobile Manipulation using Low-Cost Whole-Body Teleoperation, CoRL 2024 — 全身遥操作与静态数据联合训练Whole-body teleoperation and co-training with static data
ALOHA 2 Team (J. Aldaco et al.), ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperation, 2024 — 硬件改进、开源设计与 MuJoCo 模型Hardware improvements, open designs and a MuJoCo model
C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake & S. Song, Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots, RSS 2024 — 手持夹爪、相对轨迹动作与延迟匹配The hand-held gripper, relative-trajectory actions and latency matching
P. Wu, Y. Shentu, Z. Yi, X. Lin & P. Abbeel, GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators, 2023 — 运动学等价的低成本主臂Low-cost kinematically equivalent leader arms
A. Mandlekar et al., RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation, CoRL 2018 — 用手机远程众包采集示范Crowdsourcing demonstrations remotely with phones
F. Lin, Y. Hu, P. Sheng, C. Wen, J. You & Y. Gao, Data Scaling Laws in Imitation Learning for Robotic Manipulation, 2024 — 多样性与数量的大规模真机研究A large real-robot study of diversity vs quantity
A. Khazatsky et al., DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset, 2024 — 多地点、多采集员的大规模遥操作数据集A large teleoperation dataset across many sites and collectors
Open X-Embodiment Collaboration, Open X-Embodiment: Robotic Learning Datasets and RT-X Models, 2023 — 跨形态数据集与 RT-XThe cross-embodiment dataset and RT-X
S. Ramos et al., RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning, 2021 — episode / step 数据格式The episode / step data format
R. J. Anderson & M. W. Spong, Bilateral Control of Teleoperators with Time Delay, IEEE Trans. Automatic Control 34(5), 1989 — 有延迟的力反馈遥操作与无源性Force-feedback teleoperation under delay, and passivity
W. R. Ferrell, Remote Manipulation with Transmission Delay, IEEE Trans. Human Factors in Electronics, 1965 — 延迟下操作员的 move-and-wait 策略The operator's move-and-wait strategy under delay
P. F. Hokayem & M. W. Spong, Bilateral Teleoperation: An Historical Survey, Automatica 42(12), 2006 — 双边遥操作综述A survey of bilateral teleoperation