Part V · 第 20 章强化学习 · 20.5Part V · Ch. 20 Reinforcement Learning · 20.5

离线强化学习Offline RL

真机试错又慢又贵，还会弄坏东西，可是机器人往往已经攒下了大量日志：遥操作示教、旧控制器的运行记录、以前的实验。离线强化学习只用这样一份固定的数据集学策略，一次也不和环境交互。难点在于：Q 函数对数据里从没出现过的动作只能外推，而 $\max_a$ 偏偏会去挑这些被高估的动作，又没有新数据来纠正它。本节在浏览器里现场训练一个推箱子任务上的五种学习器：天真的离线 Q-learning 怎样把价值吹到不可能的高度、把箱子推倒；策略约束（BCQ、TD3+BC）、保守 Q-learning（CQL）和隐式 Q-learning（IQL）怎样各自让策略留在数据附近；离线 RL 什么时候比行为克隆强，以及怎样从离线训练过渡到在线微调。 Trial and error on a real robot is slow, expensive and breaks things, yet robots often already have plenty of logs: teleoperated demonstrations, runs of older controllers, past experiments. Offline reinforcement learning learns a policy from such a fixed dataset without a single new interaction. The difficulty: a Q-function can only extrapolate on actions the data never contains, and $\max_a$ seeks out exactly those overestimated actions, with no new data to correct them. This lesson trains five learners live in your browser on a box-pushing task: how naive offline Q-learning inflates values to impossible heights and tips the box over; how policy constraints (BCQ, TD3+BC), conservative Q-learning (CQL) and implicit Q-learning (IQL) each keep the policy near the data; when offline RL beats behaviour cloning; and how to move from offline training to online fine-tuning.

约 45 分钟~45 min 前置：20.2 Q-learning 与 DQN · 20.4 Actor-Critic：PPO 与 SAC · 16.2 损失、泛化与正则化 · 6.12 动态规划Prereqs: 20.2 Q-Learning & DQN · 20.4 Actor-Critic: PPO & SAC · 16.2 Losses, Generalization & Regularization · 6.12 Dynamic programming 5 个交互5 interactives

1为什么机器人需要它Why robots need this

20.4 节的 PPO 和 SAC 都要边学边试：策略产生新数据，新数据改进策略。在仿真里这没问题，一台 GPU 上可以同时跑几千个环境；可在真机上，每一次试错都花时间、要人看着，还可能摔坏机器人或打翻物体。与此同时，机器人领域的数据并不少：遥操作示教（22.3 节）、旧控制器跑过的日志、以前项目里的失败和成功记录。能不能只用这份已有的数据，不再和环境交互，就学出一个比数据里的行为更好的策略？这就是离线强化学习（offline RL，早期也叫 batch RL）。

它和22.1 节的行为克隆（BC）用的是同一种数据，区别在于目标：BC 模仿数据里的动作，最多做到数据的平均水平；离线 RL 利用数据里的奖励，挑出好的动作、把不同轨迹里好的片段拼起来，有机会超过收集数据的人或控制器。

听起来只要把 20.2 节的 DQN 或 20.4 节的 SAC 拿来，在固定的回放缓冲区上训练就行了（它们本来就是 off-policy 的）。可实际上这样训练出来的策略往往很差，甚至比随便哪个数据里的行为都差。原因是一种特殊的分布偏移（16.2 节）：学到的策略会去选数据里从未出现过的动作，Q 函数对这些动作的估计只是网络的外推，而且往往被高估。在线 RL 里，策略试一下就会发现"原来没那么好"；离线时，这个错误永远得不到纠正。本节的主线就是这个问题，以及三类对策。

全篇用同一个任务：机械臂沿桌面把一个高高的箱子推向目标线。状态 $s\in[0,1]$ 是箱子的位置（0 是起点，到 1 就成功），动作 $a\in[0,1]$ 是推的速度指令，每一步箱子前进 $0.3a$，奖励是每步 $-1$（越快到越好），折扣 $\gamma=0.9$。但推得太快（$a>0.75$）箱子会倒：回合结束，奖励 $-21$。数据来自两位操作员共 40 个回合：一位谨慎（速度约 0.2），一位熟练（约 0.48），两人都从不推得快于 0.55。最优策略是始终以 0.75 推，5 步到达；数据里最快的 0.55 需要 7 步，谨慎的速度 0.2 需要 17 步。

PPO and SAC from Lesson 20.4 both learn by trying: the policy produces new data, and the new data improves the policy. That is fine in simulation, where one GPU runs thousands of environments at once, but on a real robot every trial costs time, needs supervision, and may break the robot or knock objects over. At the same time robotics is not short of data: teleoperated demonstrations (Lesson 22.3), logs of older controllers, successes and failures from past projects. Can we learn a policy that is better than the behaviour in the data, using only that data and no further interaction? That is offline reinforcement learning (offline RL, earlier called batch RL).

It uses the same kind of data as behaviour cloning in Lesson 22.1 (BC), but with a different goal: BC imitates the actions in the data and at best reaches the data's average level; offline RL uses the rewards in the data to pick out the good actions and stitch good pieces of different trajectories together, so it has a chance to outperform whoever collected the data.

It sounds as if we could take the DQN of Lesson 20.2 or the SAC of Lesson 20.4 and train it on a fixed replay buffer (they are off-policy, after all). In practice the policy trained this way is often bad, even worse than any behaviour in the data. The reason is a special kind of distribution shift (Lesson 16.2): the learned policy chooses actions the data never contains, the Q-function's estimate for those actions is pure extrapolation by the network, and it is often too high. Online, the policy would try such an action and find out "it's not that good"; offline, the error is never corrected. This problem and three families of fixes are the thread of the lesson.

One task runs through the whole page: a robot arm pushes a tall box along a table towards a target line. The state $s\in[0,1]$ is the box position (0 is the start, reaching 1 is success), the action $a\in[0,1]$ is the push-speed command, each step moves the box by $0.3a$, the reward is $-1$ per step (the sooner the better), and the discount is $\gamma=0.9$. But pushing too fast ($a>0.75$) tips the box over: the episode ends with reward $-21$. The data come from two operators, 40 episodes in total: a cautious one (speed about 0.2) and a skilled one (about 0.48), and neither ever pushes faster than 0.55. The optimal policy pushes at 0.75 every step and arrives in 5 steps; the fastest speed in the data, 0.55, needs 7 steps, and the cautious 0.2 needs 17.

2直觉：网络在没见过的地方只能外推Intuition: where it has seen nothing, a network can only extrapolate

先把问题缩到一步：只推一次，结果 $y$ 是箱子前进的距离（$a\le0.75$ 时就是 $a$），推倒了是 $-1$。数据是 24 次推箱子，速度都在 $[0.05,0.55]$ 里。我们用 5 个随机初始化不同的小网络（ReLU MLP）去拟合 $\hat Q(a)\approx\mathbb E[y\mid a]$，再选 $\hl{f1max}{\arg\max_a\hat Q(a)}$。这就是离线 RL 里"先估计价值、再取最大"这一步的缩影。

在数据覆盖的范围里，五个网络都拟合得很好，彼此几乎一样。到了数据外面，它们只能各自外推：ReLU 网络大致会把数据里的趋势"越快越好"线性延伸出去，于是 $\arg\max$ 毫不犹豫地选 $a=1$，箱子倒了。注意网络并没有"出错"：它在有数据的地方都是对的，错的是我们拿它去回答一个数据根本没有信息的问题，而 $\max$ 恰好会找到误差最偏高的那个地方。

一个朴素的对策是悲观：既然 5 个网络在数据外分歧很大，就按下置信界 $\hl{f1lcb}{\hat\mu(a)-k\,\hat\sigma(a)}$ 来选（$\hat\mu,\hat\sigma$ 是 5 个网络的均值和标准差，21.3 节会细讲集成），不确定的地方先打折。

First shrink the problem to one step: push once, and the outcome $y$ is the distance the box travels (just $a$ when $a\le0.75$), or $-1$ if it tips. The data are 24 pushes, all with speeds in $[0.05,0.55]$. We fit $\hat Q(a)\approx\mathbb E[y\mid a]$ with 5 small networks (ReLU MLPs) that differ only in their random initialization, then choose $\hl{f1max}{\arg\max_a\hat Q(a)}$. This is the "estimate values, then take the max" step of offline RL in miniature.

Inside the range the data cover, all five networks fit well and agree almost perfectly. Outside it, each can only extrapolate: a ReLU network roughly continues the trend in the data, "faster is better", in a straight line, so $\arg\max$ picks $a=1$ without hesitation and the box tips. Note that the networks did not "make a mistake": they are right wherever there is data. The mistake is asking them a question the data hold no information about, and $\max$ is precisely the operation that finds the place where the error is most positive.

A simple fix is pessimism: since the 5 networks disagree strongly outside the data, choose by the lower confidence bound $\hl{f1lcb}{\hat\mu(a)-k\,\hat\sigma(a)}$ instead ($\hat\mu,\hat\sigma$ are the mean and standard deviation of the 5 networks; ensembles are covered in Lesson 21.3), discounting whatever is uncertain.

FIG 1外推误差：一次推箱子的价值估计Extrapolation error: value estimates for a single push

紫线是真实结果（$a>0.75$ 的红色区域里箱子会倒），琥珀点是 24 次带噪声（标准差 0.03）的推箱子记录，青色带是数据覆盖的速度范围，拖动它两端的青色圆圈可以改范围（松手后重新拟合）。蓝色细线是 5 个网络（每个 2 层 24 个 ReLU 单元，Adam 训练 500 步），蓝色粗线是它们的均值 $\hat\mu$，绿线是 $\hat\mu-k\hat\sigma$。红色竖线是直接取 $\arg\max\hat\mu$ 的选择，绿色竖线是按下置信界的选择。默认数据下，种子 1、2、3 的直接选择都是 1.00；种子 1 需要 $k\approx1.5$ 才退回数据里，种子 2 要 $k=3$，种子 3 连 $k=3$ 都不够。 The purple line is the true outcome (in the red region, $a>0.75$, the box tips); the amber dots are 24 logged pushes with noise (standard deviation 0.03); the cyan band is the range of speeds the data cover, and dragging the cyan circles at its ends changes it (the fit restarts when you let go). The thin blue lines are the 5 networks (2 layers of 24 ReLU units each, 500 Adam steps), the thick blue line is their mean $\hat\mu$, and the green line is $\hat\mu-k\hat\sigma$. The red vertical line is the plain $\arg\max\hat\mu$ choice, the green vertical line the lower-confidence-bound choice. With the default data the plain choice is 1.00 for seeds 1, 2 and 3; seed 1 needs $k\approx1.5$ to come back into the data, seed 2 needs $k=3$, and for seed 3 even $k=3$ is not enough.

试一试Try this

保持默认数据和种子，把悲观系数 $k$ 加大，直到绿色的选择回到数据范围里（不超过右端 0.05 以上），并且真实前进距离至少 0.45，而直接取最大仍落在数据外。种子 1 大约需要 $k=1.5$。Keep the default data and seed and raise the pessimism $k$ until the green choice is back inside the data (no more than 0.05 beyond the right end) with a true gain of at least 0.45, while the plain maximum is still outside. Seed 1 needs about $k=1.5$.
把数据的右端拖到 0.9 以上，让数据里也有几次推倒的记录。重新拟合后，即使 $k=0$，直接取最大也落在 0.75 附近，前进距离至少 0.6：数据覆盖是比任何技巧都可靠的解药，但离线时你没法自己去收集它。Drag the right end of the data past 0.9 so that the data also contain a few tipped pushes. After the refit, even with $k=0$ the plain maximum lands near 0.75 with a gain of at least 0.6: coverage is a more reliable cure than any trick, but offline you cannot go and collect it.

核心洞见Key insight

离线 RL 的根本困难不是"网络拟合得不好"，而是优化器会主动寻找估计误差。数据里没有的动作，估计值可以偏高也可以偏低，但 $\max$ 只会把偏高的那些挑出来。监督学习只在测试分布上出错；离线 RL 是策略自己选择去测试分布外的地方。

The fundamental difficulty of offline RL is not "the network fits badly" but that the optimizer actively seeks out estimation errors. For actions absent from the data the estimate may be too high or too low, but $\max$ picks out only the ones that are too high. Supervised learning merely fails on a shifted test distribution; in offline RL the policy itself chooses to go outside the training distribution.

3动手：天真的离线 Q-learningPlay: naive offline Q-learning

现在回到多步的推箱子任务。最直接的做法是 20.2 节的拟合 Q 迭代 / DQN：网络 $Q_\theta(s,a)$，目标网络 $Q_{\theta^-}$ 是它缓慢更新的副本，每次从固定数据集 $\mathcal D$ 里抽 32 条转移 $(s,a,r,s')$，回归到

Back to the multi-step pushing task. The most direct approach is the fitted Q-iteration / DQN of Lesson 20.2: a network $Q_\theta(s,a)$, a target network $Q_{\theta^-}$ that is a slowly updated copy of it, and minibatches of 32 transitions $(s,a,r,s')$ drawn from the fixed dataset $\mathcal D$, regressed onto

$$ \begin{gathered} y \;=\; r+\gamma\,\hl{maxa}{\max_{a'}}\,Q_{\theta^-}(s',a'),\qquad \min_\theta\ \mathbb E_{\hl{data}{(s,a,r,s')\sim\mathcal D}}\big[(Q_\theta(s,a)-y)^2\big],\\ \pi(s)=\arg\max_a Q_\theta(s,a). \end{gathered} $$

动作是连续的，这里的 $\max$ 和 $\arg\max$ 在 21 个候选速度 $0,0.05,\dots,1$ 上取。问题出在 $\hl{maxa}{\max_{a'}}$：它会查询 $a'>0.55$ 或 $a'<0.05$ 这些数据里没有的动作。这些地方的 $Q$ 只是外推值，一旦偏高，它就进入目标 $y$，再通过下一次回归传给前一个状态，一步步往回累积。在线学习时，策略会真的去执行这些动作、看到真实的后果；离线时，没有任何数据会把它拉回来。

下图左边是 $Q_\theta(s,a)$ 的热力图（横轴箱子位置，纵轴推速），右边比较两个数：学习器对起点 $s=0$ 的自我评估 $Q(0,\pi(0))$，和它的策略在真实环境里从 $s=0$ 跑一遍得到的实际回报。每一步的奖励都不超过 $-1$，所以任何 $Q$ 都不可能高于 $-1$；最优策略的回报是 $V^\star(0)=-4.10$。

Actions are continuous; here $\max$ and $\arg\max$ are taken over 21 candidate speeds $0,0.05,\dots,1$. The trouble is $\hl{maxa}{\max_{a'}}$: it queries actions such as $a'>0.55$ or $a'<0.05$ that the data never contain. There, $Q$ is only an extrapolated value; if it is too high it enters the target $y$, is passed back to the previous state by the next regression, and accumulates step by step. Online, the policy would actually execute those actions and see the real consequences; offline, no data ever pull the estimate back.

On the left of the figure is a heat map of $Q_\theta(s,a)$ (box position across, push speed up); on the right two numbers are compared: the learner's own estimate of the start, $Q(0,\pi(0))$, and the actual return its policy gets when run once from $s=0$ in the real environment. Every reward is at most $-1$, so no $Q$ can ever exceed $-1$; the optimal policy's return is $V^\star(0)=-4.10$.

FIG 2天真的离线 Q-learning：价值膨胀与推倒的箱子Naive offline Q-learning: inflated values and a tipped box

左："数值"视图里，格子越蓝 $Q_\theta$ 越高（从 $-12$ 到 $-1$），红色格子是 $Q_\theta>-0.5$，明显超过了不可能逾越的上限 $-1$；"误差"视图画 $Q_\theta-Q^\star$：越红高估越多，越青低估越多。紫点是 527 条记录的 $(s,a)$，在线模式下新收集的最近 300 条画成青点。琥珀色曲线是当前策略 $\pi(s)$，红色虚线 $a=0.75$ 之上箱子会倒（它也正是最优的速度）。右上：蓝线是自我评估 $Q(0,\pi(0))$，绿线是实际回报，虚线是 $V^\star(0)$，每 100 次迭代记录一次。下方条带：按"运行策略"让机械臂按当前策略推一次。每帧按时间预算跑若干次迭代（约 7 ms），速度因电脑而异，但同一迭代次数下结果完全一样（种子固定）。 Left: in the "value" view, bluer cells mean higher $Q_\theta$ (from $-12$ to $-1$), and red cells have $Q_\theta>-0.5$, clearly above the impossible ceiling $-1$; the "error" view shows $Q_\theta-Q^\star$: redder means more overestimated, more cyan means more underestimated. Purple dots are the $(s,a)$ of the 527 logged transitions; in online mode the most recent 300 newly collected ones are drawn as cyan dots. The amber curve is the current policy $\pi(s)$; above the red dashed line $a=0.75$ the box tips (and that is also the optimal speed). Top right: the blue line is the own estimate $Q(0,\pi(0))$, the green line the actual return, the dashed line $V^\star(0)$, recorded every 100 iterations. The strip below: press "Run policy" to watch the arm push once with the current policy. Each frame runs as many iterations as fit in about 7 ms, so the speed depends on your computer, but the result after a given number of iterations is always the same (the seeds are fixed).

试一试Try this

按"训练"（离线）至少 1000 次迭代，直到自我评估比实际回报高出 5 以上。默认种子在第 1000 次迭代时，学习器认为起点值 $+1.6$（比任何可能的值都高），实际回报却是 $-9.85$：它在原地不动。Press "Train" (offline) for at least 1000 iterations, until the own estimate exceeds the actual return by more than 5. With the default seed at iteration 1000 the learner values the start at $+1.6$ (higher than any possible value), while the actual return is $-9.85$: it stands still.
离线训练 1000 次以上、策略失败时（推倒箱子或停在原地），按"运行策略"看机械臂怎么做。默认种子在第 1500 次迭代左右开始以 1.00 的速度推，第一步就把箱子推倒；换个种子，失败的方式可能变成"一动不动"：误差往哪边偏是随机的，被 $\max$ 挑中是必然的。After more than 1000 offline iterations, while the policy fails (tips the box or stays put), press "Run policy" and watch the arm. With the default seed, from around iteration 1500 it pushes at 1.00 and tips the box on the first step; with another seed the failure may instead be "not moving at all": which way the error leans is random, that $\max$ finds it is not.
切换到"在线"（每次迭代用当前策略真实走一步，20% 的概率随机选速度），继续训练，直到策略在 8 步以内到达目标。默认种子大约需要 1100 到 1600 次在线迭代，看看计数器上一共推倒了多少个箱子（通常在 50 到 150 之间）。Switch to "online" (each iteration takes one real step with the current policy, choosing a random speed 20% of the time) and keep training until the policy reaches the target in at most 8 steps. With the default seed this takes about 1100 to 1600 online iterations; look at how many boxes the counter says were tipped on the way (usually between 50 and 150).

核心洞见Key insight

在线 RL 有一个自动纠错回路：高估某个动作 → 去执行它 → 看到真实回报 → 估计被拉下来。离线 RL 切断了这个回路，于是 20.2 节讲过的最大化偏差不再是一个小偏差，而是会经由自举一步步累积、没有上限的错误。在线模式能修好它，代价是真实的失败（推倒的箱子）：这正是我们一开始想用离线数据避免的东西。

Online RL has an automatic error-correction loop: overestimate an action → execute it → see the real return → the estimate comes down. Offline RL cuts that loop, so the maximization bias of Lesson 20.2 is no longer a small bias but an error that accumulates through bootstrapping with no ceiling. Online mode repairs it at the price of real failures (tipped boxes), exactly what we wanted to avoid by using offline data in the first place.

4三类对策：约束策略、压低价值、只在样本内学习Three fixes: constrain the policy, push values down, learn only in-sample

既然问题出在"查询数据外的动作"，对策也围绕它展开，大致分三类：

策略约束 / 行为正则：只允许策略选择离数据中行为 $\pi_\beta$ 不远的动作。BCQ（Fujimoto 等，2019）先用一个生成模型拟合 $\pi_\beta$，只在它采样出的候选动作里（加小扰动）取最大；TD3+BC（Fujimoto 与 Gu，2021）更简单：在 TD3（20.4 节）的 actor 目标里加一项 BC 损失，$\max_\pi\ \lambda Q(s,\pi(s))-\hl{bcterm}{(\pi(s)-a)^2}$，$\lambda=\alpha/\overline{|Q|}$。
保守价值：直接修改 Q 的训练目标，把数据外动作的值压下去。CQL（Kumar 等，2020）在 TD 损失外加 $\alpha\big(\log\sum_a e^{Q(s,a)}-Q(s,a_{\mathcal D})\big)$：压低所有动作的"软最大值"，同时抬高数据里实际出现过的动作。
样本内学习：干脆不查询数据外的动作。IQL（Kostrikov 等，2022）用一个状态价值网络 $V(s)$ 代替 $\max_{a'}Q(s',a')$，$V$ 只用数据里的 $(s,a)$ 通过期望分位回归（expectile，第 5 节）学成"数据里好动作的价值"，最后用优势加权的 BC 提取策略。整个过程中 Q 只在数据里的动作上被训练和读取。

下图在同一份数据、同一个网络结构上训练这四种学习器（BC 作为基线）。换方法或改滑块会从头重新训练。

Since the problem is "querying actions outside the data", the fixes revolve around it, in roughly three families:

Policy constraints / behaviour regularization: only let the policy choose actions close to the behaviour $\pi_\beta$ in the data. BCQ (Fujimoto et al., 2019) fits a generative model of $\pi_\beta$ and takes the max only among candidate actions sampled from it (plus small perturbations); TD3+BC (Fujimoto & Gu, 2021) is simpler: it adds a BC loss to the actor objective of TD3 (Lesson 20.4), $\max_\pi\ \lambda Q(s,\pi(s))-\hl{bcterm}{(\pi(s)-a)^2}$ with $\lambda=\alpha/\overline{|Q|}$.
Conservative values: change Q's training objective so that values of out-of-data actions are pushed down. CQL (Kumar et al., 2020) adds $\alpha\big(\log\sum_a e^{Q(s,a)}-Q(s,a_{\mathcal D})\big)$ to the TD loss: it pushes down a "soft maximum" over all actions while pushing up the actions that actually occur in the data.
In-sample learning: never query an action outside the data at all. IQL (Kostrikov et al., 2022) replaces $\max_{a'}Q(s',a')$ by a state-value network $V(s)$ that is learned only from the $(s,a)$ in the data by expectile regression (Section 5) to mean "the value of the good actions in the data", and finally extracts a policy by advantage-weighted BC. Throughout, Q is trained and read only at actions that occur in the data.

The figure trains these four learners (BC as the baseline) on the same data with the same network architecture. Changing the method or a slider restarts training from scratch.

FIG 3同一份数据，四种学习器One dataset, four learners

画法同 FIG 2。BC 不学 Q 函数，所以它的热力图是空的。TD3+BC 和 IQL 的策略是一个单独的 actor 网络，CQL 与 FIG 2 一样对 Q 取 $\arg\max$。CQL 的 $\log\sum e^{Q}$ 用每个状态 8 个均匀随机动作估计。IQL 的 $\tau$ 固定为 0.9（FIG 4 专门看它），滑块调的是策略提取的温度 $\beta$。下方记分表记录每种设置训练到 2000 次以上时的结果。用默认种子训练 3000 次：BC 14 步；TD3+BC（$\alpha=0.5$）10 步；CQL（$\alpha=0.3$）10 步，$\alpha=0.1$ 时 8 步；IQL（$\beta=10$）8 步。 Drawn as in FIG 2. BC learns no Q-function, so its heat map is empty. TD3+BC and IQL act with a separate actor network; CQL takes $\arg\max$ of Q as in FIG 2. CQL's $\log\sum e^{Q}$ is estimated with 8 uniformly random actions per state. IQL's $\tau$ is fixed at 0.9 (FIG 4 is all about it); its slider sets the policy-extraction temperature $\beta$. The scoreboard below records each setting's result once it has trained for 2000 iterations or more. With the default seed after 3000 iterations: BC 14 steps; TD3+BC ($\alpha=0.5$) 10 steps; CQL ($\alpha=0.3$) 10 steps, and 8 steps at $\alpha=0.1$; IQL ($\beta=10$) 8 steps.

试一试Try this

找一种方法和设置，使策略不推倒箱子、在 8 步以内到达（BC 要 14 步；数据里最快的 0.55 要 7 步）。提示：IQL 默认设置训练约 2000 次，或 CQL 取 $\alpha=0.1$ 训练约 2000 次。Find a method and setting whose policy reaches the target within 8 steps without tipping the box (BC needs 14; the fastest speed in the data, 0.55, needs 7). Hint: IQL at its default for about 2000 iterations, or CQL with $\alpha=0.1$ for about 2000.
把 CQL 的 $\alpha$ 调到 1 或以上，训练至少 2000 次：策略比 BC 还慢（超过 14 步，默认种子是 18 步）。太保守的 CQL 会退化成"选数据里最常见的动作"，而最常见的是谨慎操作员的慢速。Set CQL's $\alpha$ to 1 or more and train for at least 2000 iterations: the policy is slower than BC (more than 14 steps; 18 with the default seed). Too conservative a CQL degenerates into "choose the most common action in the data", and the most common is the cautious operator's slow speed.
把 TD3+BC 的 $\alpha$ 调到 2.5 或以上，训练至少 1500 次：在默认种子上策略失败（停在原地）。BC 项太弱时，actor 会顺着 Q 的外推梯度走出数据，又回到了 FIG 2 的问题。Set TD3+BC's $\alpha$ to 2.5 or more and train for at least 1500 iterations: with the default seed the policy fails (stays put). With too weak a BC term the actor follows Q's extrapolated gradient out of the data, and we are back to the problem of FIG 2.

核心洞见Key insight

切到"误差"视图会发现：三种对策都没有把数据外的 Q 修正确，倒箱子区域的值仍然高估了大约 7 到 20。它们做的是另一件事：不让策略去利用这些错误。TD3+BC 把 actor 拴在数据上，CQL 把数据外的值相对压低，IQL 根本不去读它们。每一种都有一个旋钮在"像 BC 一样稳"与"像 RL 一样敢"之间取舍，而且旋钮的合适位置要看数据。

Switch to the "error" view and you will see that none of the three fixes makes Q correct outside the data: the tipping region is still overestimated by roughly 7 to 20. They do something else: they stop the policy from exploiting those errors. TD3+BC tethers the actor to the data, CQL pushes out-of-data values down relative to the data, and IQL never reads them. Each has a knob that trades "as safe as BC" against "as bold as RL", and where the knob belongs depends on the data.

5第二、第三视角：期望分位回归，以及离线 RL 与 BCSecond and third views: expectile regression, and offline RL versus BC

IQL 怎样在不取 max 的情况下"取 max"How IQL takes a "max" without taking a max

IQL 想要的是"在数据里出现过的动作中，最好的那个值"，即 $\max_{a\in\operatorname{supp}\pi_\beta(\cdot\mid s)}Q(s,a)$，但不能真的对 $a$ 取最大（那会查询数据外的动作）。它的做法是对同一状态下数据里各个动作的 $Q(s,a_i)$ 做期望分位回归：最小化非对称平方损失

IQL wants "the value of the best action among those that occur in the data", $\max_{a\in\operatorname{supp}\pi_\beta(\cdot\mid s)}Q(s,a)$, without actually maximizing over $a$ (which would query actions outside the data). It does so by expectile regression on the values $Q(s,a_i)$ of the logged actions at a state: minimize the asymmetric squared loss

$$ \hl{loss}{L_\tau(v)}=\frac1n\sum_i\big|\tau-\mathbb 1(u_i<0)\big|\,u_i^2,\qquad u_i=Q(s,a_i)-v,\qquad \hl{vtau}{V_\tau(s)}=\arg\min_v L_\tau(v). $$

$\tau=0.5$ 时它就是普通的平方损失，最优解是均值；$\tau$ 越接近 1，$v$ 低于样本（$u_i>0$）受的罚越重，$V_\tau$ 就越往上推，逼近数据里的最大值，却永远不会超过它，因为它只是这些样本的一个加权平均。下图用 40 个示意的 $Q$ 值代表推箱子任务某一状态下数据中各动作的价值（数值范围取自这个任务，七成来自谨慎操作员，三成来自熟练操作员）。

At $\tau=0.5$ it is the ordinary squared loss and the minimizer is the mean; as $\tau$ approaches 1, $v$ being below a sample ($u_i>0$) is penalized more and more, so $V_\tau$ rises towards the largest value in the data but never exceeds it, because it is only a weighted average of those samples. The figure uses 40 illustrative $Q$-values to stand for the values of the logged actions at one state of the pushing task (in the range of that task; seven in ten from the cautious operator, three in ten from the skilled one).

FIG 4期望分位：从均值滑向样本内最大值Expectiles: from the mean towards the in-sample max

数据中动作的 Q 值Q-VALUES OF LOGGED ACTIONS

损失 L_τ(v)：拖动琥珀色圆圈LOSS L_τ(v) · drag the amber circle

左：紫点是 40 个 $Q(s,a_i)$（纵向错开只为看清），虚线是均值，绿线是 $V_\tau$，蓝线是数据里的最大值，红线是对所有动作取 $\max$ 时可能得到的外推值（一个示意值 $-2.3$，代表 FIG 2 里那种从未尝试过的动作）。右：青色曲线是 $L_\tau(v)$，琥珀色圆圈是你选的 $v$，绿线标出最小值的位置。这些数值的均值是 $-7.19$，最大值 $-6.00$；$\tau=0.9$ 时 $V_\tau=-6.60$，$\tau=0.99$ 时 $-6.21$。 Left: the purple dots are the 40 values $Q(s,a_i)$ (spread vertically only for visibility), the dashed line is their mean, the green line is $V_\tau$, the blue line the largest value in the data, and the red line an extrapolated value that a $\max$ over all actions might return (an illustrative $-2.3$, standing for a never-tried action like those in FIG 2). Right: the cyan curve is $L_\tau(v)$, the amber circle is your choice of $v$, and the green line marks the minimizer. The values have mean $-7.19$ and maximum $-6.00$; $V_\tau=-6.60$ at $\tau=0.9$ and $-6.21$ at $\tau=0.99$.

试一试Try this

把 $\tau$ 调到 0.8 以上，然后拖动琥珀色圆圈去找损失的最低点（与 $V_\tau$ 相差不到 0.05）。注意最低点在均值的右边：低估一个样本的代价是高估的 $\tau/(1-\tau)$ 倍。Set $\tau$ above 0.8, then drag the amber circle to the bottom of the loss (within 0.05 of $V_\tau$). Notice the minimum lies to the right of the mean: underestimating a sample costs $\tau/(1-\tau)$ times as much as overestimating it.
把 $\tau$ 推到 0.99 以上：$V_\tau$ 离数据里的最大值不到 0.25，但仍在它左边，离红线（外推值）还远。Push $\tau$ to 0.99 or more: $V_\tau$ gets within 0.25 of the largest value in the data, yet stays to its left, still far from the red line (the extrapolated value).

离线 RL 什么时候比 BC 强？When does offline RL beat BC?

IQL 的策略提取 $\max_\pi\mathbb E_{\mathcal D}\big[e^{\beta(Q(s,a)-V(s))}\log\pi(a\mid s)\big]$ 本身就是加权的 BC：$\beta=0$ 时就是 BC（FIG 3 里把 IQL 的 $\beta$ 拉到 0，结果与 BC 一模一样，14 步）。另一个更简单的基线是过滤式 BC：只保留回报最高的 20% 回合，再做 BC。如果好行为都集中在某些完整的回合里，过滤式 BC 就够了；离线 RL 的独特之处在于能逐条转移地判断好坏，把散落在不同回合中的好片段拼起来。下图比较两种数据：一种是两位操作员各自完整地推完一个回合；另一种是同一位操作员在每一步都随机地快或慢。

IQL's policy extraction $\max_\pi\mathbb E_{\mathcal D}\big[e^{\beta(Q(s,a)-V(s))}\log\pi(a\mid s)\big]$ is itself a weighted BC: at $\beta=0$ it is BC (drag IQL's $\beta$ to 0 in FIG 3 and you get exactly BC's result, 14 steps). An even simpler baseline is filtered BC: keep only the 20% of episodes with the highest return and run BC on them. If the good behaviour comes in whole episodes, filtered BC suffices; what is unique to offline RL is that it judges every transition separately and can stitch together good pieces scattered across different episodes. The figure compares two kinds of data: in one, each of the two operators pushes a whole episode; in the other, a single operator is randomly fast or slow at every step.

FIG 5BC、过滤式 BC 与 IQL：数据质量决定差距BC, filtered BC and IQL: the data decide the gap

数据 (s, a) 与学到的策略DATA (s, a) AND LEARNED POLICIES

到达目标的步数（越少越好）STEPS TO THE TARGET (fewer is better)

左：青点是谨慎速度的转移，琥珀点是熟练速度的转移；三条曲线是学到的策略 $\pi(s)$：红 BC（2000 次迭代），紫过滤式 BC（只用回报最高的 20% 回合，2000 次），绿 IQL（$\tau=0.9$，$\beta=10$，3000 次）。右：三者从 $s=0$ 出发到达目标的步数，虚线是数据里最快速度 0.55 的 7 步，绿色点线是最优的 5 步。默认种子的结果：整回合数据、熟练比例 0.3 时为 14 / 7 / 8 步；每步切换、比例 0.3 时为 12 / 10 / 8 步；整回合、比例为 1（全是熟练数据）时为 7 / 7 / 8 步。 Left: cyan dots are transitions at the cautious speed, amber dots at the skilled speed; the three curves are the learned policies $\pi(s)$: red BC (2000 iterations), purple filtered BC (only the top 20% of episodes by return, 2000 iterations), green IQL ($\tau=0.9$, $\beta=10$, 3000 iterations). Right: the number of steps each needs from $s=0$ to the target; the dashed line marks 7 steps (the fastest speed in the data, 0.55) and the dotted green line 5 (optimal). Results with the default seed: whole episodes at skilled fraction 0.3, 14 / 7 / 8 steps; switching every step at 0.3, 12 / 10 / 8 steps; whole episodes at fraction 1 (all skilled), 7 / 7 / 8 steps.

试一试Try this

选"整回合"，熟练比例取 0.2 到 0.5 之间，按"运行"：过滤式 BC 和 IQL 相差不超过 1 步，都比 BC 快至少 5 步。好行为整回合地出现时，挑回合就够了。Choose "whole episodes", set the skilled fraction between 0.2 and 0.5, and press "Run": filtered BC and IQL are within 1 step of each other, and both are at least 5 steps faster than BC. When good behaviour comes in whole episodes, picking episodes is enough.
选"每步切换"，比例取 0.1 到 0.4 之间，再运行：IQL 比过滤式 BC 至少快 2 步。每个回合都是快慢混杂的，按回合过滤挑不出好动作，按转移加权才行。Choose "switching every step", a fraction between 0.1 and 0.4, and run again: IQL is at least 2 steps faster than filtered BC. Every episode mixes fast and slow pushes, so filtering by episode cannot isolate the good actions; weighting each transition can.

一个问题，三个视角One problem, three views

分布偏移视角：学到的策略的状态–动作分布 $d^\pi$ 离开了数据分布 $d^{\pi_\beta}$，16.2 节的重要性权重 $d^\pi/d^{\pi_\beta}$ 在数据外是无穷大，所以任何对 $Q^\pi$ 的估计都没有依据。不确定性视角：数据外的 $Q$ 不是"错的"，而是"未知的"；离线 RL 要对未知保持悲观（FIG 1 的下置信界，CQL 的压低）。约束优化视角：在"离数据不远"的约束下最大化回报，BC 是约束最紧的一端，天真的离线 Q-learning 是没有约束的另一端，各种方法的旋钮就是约束的松紧。

Distribution-shift view: the learned policy's state–action distribution $d^\pi$ leaves the data distribution $d^{\pi_\beta}$; the importance weight $d^\pi/d^{\pi_\beta}$ of Lesson 16.2 is infinite outside the data, so no estimate of $Q^\pi$ there has any support. Uncertainty view: Q outside the data is not "wrong" but "unknown", and offline RL must be pessimistic about the unknown (the lower confidence bound of FIG 1, CQL's push-down). Constrained-optimization view: maximize return subject to "staying near the data"; BC is the tightest end of the constraint, naive offline Q-learning the unconstrained end, and each method's knob sets how tight it is.

6再看数学The math

提示：把鼠标悬停在公式中带底色的符号上，对应的图元素会被高亮（FIG 1 的曲线与数据点，FIG 2、FIG 3 的数据点、策略曲线、倒箱区域与误差视图，FIG 4 的损失曲线与 $V_\tau$）。Tip: hover the highlighted symbols in the formulas; the matching element lights up (FIG 1's curves and data points; the data points, policy curve, tipping region and error view of FIG 2 and FIG 3; FIG 4's loss curve and $V_\tau$).

问题设定The setting

MDP 与 20.1 节相同：状态 $s$、动作 $a$、转移 $P(s'\mid s,a)$、奖励 $r$、折扣 $\gamma$。区别只在于我们拿到的是一份固定数据集 $\hl{data}{\mathcal D=\{(s_i,a_i,r_i,s_i')\}}$，由某个（通常未知、可能是多个人或控制器混合的）行为策略 $\pi_\beta$ 收集，目标仍是最大化 $J(\pi)=\mathbb E_\pi\big[\sum_t\gamma^tr_{t+1}\big]$，但训练中不能再收集任何数据。推箱子任务里，会推倒箱子的动作 $\hl{tip}{a>0.75}$ 正是数据里从未出现过的。

The MDP is the one of Lesson 20.1: states $s$, actions $a$, transitions $P(s'\mid s,a)$, reward $r$, discount $\gamma$. The only difference is that we are handed a fixed dataset $\hl{data}{\mathcal D=\{(s_i,a_i,r_i,s_i')\}}$ collected by some behaviour policy $\pi_\beta$ (usually unknown, possibly a mixture of several people or controllers). The goal is still to maximize $J(\pi)=\mathbb E_\pi\big[\sum_t\gamma^tr_{t+1}\big]$, but no data may be collected during training. In the pushing task, the actions that tip the box, $\hl{tip}{a>0.75}$, are exactly the ones the data never contain.

外推误差为什么会累积Why extrapolation error accumulates

设每次回归后，$\hat Q$ 与真实贝尔曼目标 $\mathcal T\hat Q$ 在所有 $(s,a)$ 上相差至多 $\delta$（最大范数）。贝尔曼最优算子 $\mathcal T$ 是 $\gamma$-压缩（6.12 节），于是

Suppose that after each regression $\hat Q$ differs from its true Bellman target $\mathcal T\hat Q$ by at most $\delta$ over all $(s,a)$ (in the max norm). The Bellman optimality operator $\mathcal T$ is a $\gamma$-contraction (Lesson 6.12), so

$$ \begin{gathered} \hl{over}{\|\hat Q-Q^\star\|_\infty}\le\|\hat Q-\mathcal T\hat Q\|_\infty+\|\mathcal T\hat Q-\mathcal TQ^\star\|_\infty\le\delta+\gamma\|\hat Q-Q^\star\|_\infty \\ \Longrightarrow\ \ \|\hat Q-Q^\star\|_\infty\le\frac{\delta}{1-\gamma}. \end{gathered} $$

在线学习时，策略会访问的地方就有数据，$\delta$ 是一个统计误差，随数据增多而变小。离线时，$\delta$ 在数据外的动作上没有任何约束，可以是网络外推出的任意大小；而 $\hl{maxa}{\max_{a'}}$ 让 $\hat Q$ 的每一个目标都包含其中最偏高的那个：$\mathbb E[\max_a(Q+\varepsilon_a)]\ge\max_a Q$（20.2 节的最大化偏差）。上界里的 $1/(1-\gamma)$ 说明误差会沿时间放大：$\gamma=0.9$ 时最多 10 倍。FIG 2 里的价值膨胀就是这样来的。

Online, there is data wherever the policy goes and $\delta$ is a statistical error that shrinks as data accumulate. Offline, $\delta$ on actions outside the data is unconstrained: whatever the network extrapolates. And $\hl{maxa}{\max_{a'}}$ makes every target of $\hat Q$ contain the most positive of those errors: $\mathbb E[\max_a(Q+\varepsilon_a)]\ge\max_a Q$ (the maximization bias of Lesson 20.2). The factor $1/(1-\gamma)$ in the bound says the error is amplified over time: up to 10 times at $\gamma=0.9$. That is where the value inflation in FIG 2 comes from.

策略约束：BCQ 与 TD3+BCPolicy constraints: BCQ and TD3+BC

一般形式是带约束的策略改进：$\max_\pi\mathbb E_{s\sim\mathcal D}\big[\mathbb E_{a\sim\pi}Q(s,a)\big]$ s.t. $D\big(\pi(\cdot\mid s)\,\|\,\pi_\beta(\cdot\mid s)\big)\le\epsilon$，$D$ 可以是 KL、MMD（BEAR，Kumar 等，2019）或"只在 $\pi_\beta$ 的样本中选"（BCQ）。TD3+BC 把约束换成罚项，确定性 actor 的目标是

The general form is constrained policy improvement: $\max_\pi\mathbb E_{s\sim\mathcal D}\big[\mathbb E_{a\sim\pi}Q(s,a)\big]$ s.t. $D\big(\pi(\cdot\mid s)\,\|\,\pi_\beta(\cdot\mid s)\big)\le\epsilon$, where $D$ can be a KL divergence, an MMD (BEAR, Kumar et al., 2019) or "choose only among samples of $\pi_\beta$" (BCQ). TD3+BC replaces the constraint by a penalty; the deterministic actor maximizes

$$ \max_\pi\ \mathbb E_{(s,a)\sim\mathcal D}\Big[\lambda\,Q\big(s,\hl{pi}{\pi(s)}\big)-\hl{bcterm}{\big(\pi(s)-a\big)^2}\Big],\qquad \lambda=\frac{\alpha}{\frac1N\sum_i\big|Q(s_i,\pi(s_i))\big|}. $$

除以 $\overline{|Q|}$ 让 $\alpha$ 与奖励的尺度无关（论文默认 $\alpha=2.5$，并把状态标准化）。但 $\alpha$ 真正平衡的是 $\partial Q/\partial a$ 和 BC 项的梯度，它的合适值仍取决于任务：在推箱子任务里 Q 对动作很敏感，$\alpha=0.5$ 较好，2.5 已经太大（FIG 3）。FIG 3 的实现还把分母限制为至少 1，以免训练初期 $Q\approx0$ 时 $\lambda$ 过大，并且省略了 TD3 的双 Q 和目标策略平滑。$\alpha\to0$ 时就是 BC，$\alpha\to\infty$ 时就是不加约束的离线 TD3。

Dividing by $\overline{|Q|}$ makes $\alpha$ independent of the reward scale (the paper's default is $\alpha=2.5$, with normalized states). But what $\alpha$ really balances is $\partial Q/\partial a$ against the gradient of the BC term, so its right value still depends on the task: in the pushing task Q is very sensitive to the action, $\alpha=0.5$ works well and 2.5 is already too much (FIG 3). FIG 3's implementation also keeps the denominator at least 1, so that $\lambda$ does not blow up early in training while $Q\approx0$, and omits TD3's twin critics and target-policy smoothing. As $\alpha\to0$ it is BC; as $\alpha\to\infty$ it is unconstrained offline TD3.

保守 Q-learning（CQL）Conservative Q-learning (CQL)

CQL 的 Q 损失是

CQL's Q loss is

$$ \min_\theta\ \alpha\,\mathbb E_{s\sim\mathcal D}\Big[\log\!\int\! e^{Q_\theta(s,a)}\,da-\mathbb E_{a\sim\pi_\beta}Q_\theta(s,a)\Big]+\tfrac12\,\mathbb E_{\mathcal D}\big[(Q_\theta(s,a)-y)^2\big]. $$

第一项对 $Q(s,\cdot)$ 的梯度是 $\alpha\big(\operatorname{softmax}Q(s,\cdot)-\pi_\beta(\cdot\mid s)\big)$：值偏高、但在数据里少见的动作（softmax 权重大于 $\pi_\beta$）被压低，常见的动作被抬高。若只看这一项（$\alpha\to\infty$），平衡点是 $\operatorname{softmax}Q=\pi_\beta$，即 $Q(s,a)=\log\pi_\beta(a\mid s)+c(s)$：$\arg\max$ 变成了数据中最常见的动作。这就是 FIG 3 里 $\alpha\ge1$ 的 CQL 比 BC 还慢的原因：七成数据来自谨慎的操作员。论文证明在表格、精确备份的情形下，足够大的 $\alpha$ 使学到的策略价值成为真实价值的下界；用网络和采样近似时只能说"大致悲观"，FIG 3 的自我评估有时略低于、有时略高于实际回报。

The gradient of the first term with respect to $Q(s,\cdot)$ is $\alpha\big(\operatorname{softmax}Q(s,\cdot)-\pi_\beta(\cdot\mid s)\big)$: actions whose value is high but which are rare in the data (softmax weight above $\pi_\beta$) are pushed down, common ones pushed up. Looking at this term alone ($\alpha\to\infty$), the balance point is $\operatorname{softmax}Q=\pi_\beta$, i.e. $Q(s,a)=\log\pi_\beta(a\mid s)+c(s)$: $\arg\max$ becomes the most common action in the data. That is why CQL with $\alpha\ge1$ in FIG 3 is slower than BC: seven tenths of the data come from the cautious operator. The paper proves that with tables and exact backups a large enough $\alpha$ makes the learned policy value a lower bound on the true value; with networks and sampling it is only "roughly pessimistic", and in FIG 3 the own estimate is sometimes slightly below and sometimes slightly above the actual return.

隐式 Q-learning（IQL）Implicit Q-learning (IQL)

IQL 的三个损失都只用数据里的 $(s,a)$：

All three of IQL's losses use only the $(s,a)$ in the data:

$$ \begin{aligned} L_V&=\mathbb E_{(s,a)\sim\mathcal D}\big[\hl{loss}{L^\tau_2}\big(Q_{\theta^-}(s,a)-V(s)\big)\big],\qquad L_2^\tau(u)=|\tau-\mathbb 1(u<0)|\,u^2,\\ L_Q&=\mathbb E_{(s,a,r,s')\sim\mathcal D}\big[\big(r+\gamma\,\hl{vtau}{V(s')}-Q_\theta(s,a)\big)^2\big],\\ L_\pi&=-\,\mathbb E_{(s,a)\sim\mathcal D}\big[\min\!\big(e^{\beta(Q_{\theta^-}(s,a)-V(s))},\,100\big)\log\pi(a\mid s)\big]. \end{aligned} $$

$\tau\to1$ 时 $V(s)\to\max_{a\in\operatorname{supp}\pi_\beta(\cdot\mid s)}Q(s,a)$，于是 $L_Q$ 近似地学到"在数据支撑内行动的最优 Q"（在确定性转移下精确成立；随机转移下，过大的 $\tau$ 会把"运气好的转移"也当成动作的功劳，所以论文用 0.7 到 0.9）。$\tau=0.5$ 时 $V$ 是 $\pi_\beta$ 下的平均，IQL 退化为"一步"方法：评估行为策略，再做一次改进（Brandfonbrener 等，2021）。策略提取是优势加权回归（AWR，Peng 等，2019）：$\beta=0$ 是 BC，$\beta$ 越大越只模仿优势大的动作。FIG 3 用确定性策略和平方误差代替 $-\log\pi$，相当于固定方差的高斯策略。

As $\tau\to1$, $V(s)\to\max_{a\in\operatorname{supp}\pi_\beta(\cdot\mid s)}Q(s,a)$, so $L_Q$ approximately learns "the optimal Q when acting within the support of the data" (exactly so for deterministic transitions; with stochastic transitions too large a $\tau$ credits the action with lucky transitions, which is why the paper uses 0.7 to 0.9). At $\tau=0.5$, $V$ is the average under $\pi_\beta$ and IQL reduces to a "one-step" method: evaluate the behaviour policy, then improve once (Brandfonbrener et al., 2021). Policy extraction is advantage-weighted regression (AWR; Peng et al., 2019): $\beta=0$ is BC, and larger $\beta$ imitates only the actions with large advantage. FIG 3 uses a deterministic policy and squared error in place of $-\log\pi$, which amounts to a Gaussian policy with fixed variance.

从离线到在线From offline to online

离线训练出的策略最多只能做到"数据支撑内的最好"：推箱子任务里是 0.55 的 7 步，而不是 0.75 的 5 步。要更进一步，只能回到环境里试。直接把离线策略接上在线 RL 有两个常见问题：保守方法（如 CQL）的 Q 被压得过低，一开始收集到的真实回报比它"想象的"高，Q 被大幅修正时策略会先变差一段（"遗忘"一段，再学回来），Cal-QL（Nakamoto 等，2023）通过校准保守值、使其不低于行为策略的价值来缓解；而不加约束的方法又会去试那些被高估的危险动作（FIG 2 的在线模式，代价是推倒的箱子）。实践中常见的做法：用同一种算法继续训练（IQL、AWAC 在微调时表现平稳），把离线数据留在回放缓冲区里与在线数据按比例混合采样（RLPD，Ball 等，2023），并在真机上限制探索的幅度。

An offline-trained policy can at best be "the best within the data's support": 7 steps at 0.55 in the pushing task, not 5 steps at 0.75. To go further one has to try things in the environment again. Plugging an offline policy straight into online RL has two common problems. A conservative method such as CQL has Q pushed too low, so the first real returns are higher than it "imagined"; while Q is being corrected by a large amount the policy first gets worse for a while ("unlearning", then relearning), which Cal-QL (Nakamoto et al., 2023) mitigates by calibrating the conservative values so they do not fall below the behaviour policy's value. An unconstrained method, on the other hand, goes and tries the overestimated dangerous actions (FIG 2's online mode, paid for in tipped boxes). Common practice: keep training with the same algorithm (IQL and AWAC fine-tune smoothly), keep the offline data in the replay buffer and sample it in a fixed ratio with the online data (RLPD; Ball et al., 2023), and limit how far exploration may go on a real robot.

7机器人中的它In robotics

大规模抓取：QT-Opt。Kalashnikov 等（CoRL 2018）在多台真实机械臂上收集了数十万次抓取尝试，大部分训练数据来自以前的策略（离线），只有一部分来自当前策略的在线收集，用连续动作的 Q-learning（用 CEM 在动作上取最大，6.11 节）学出基于视觉的闭环抓取。它说明了离线数据的价值，也说明了在动作上取最大的难处。
从多任务机器人数据预训练。PTR（Kumar 等，RSS 2023）用 CQL 在一个大规模多任务机器人数据集上预训练，再用少量新任务的示教微调，学新任务所需的数据显著少于从头训练。Q-Transformer（Chebotar 等，CoRL 2023）把每个动作维度离散化、逐维自回归地输出 Q 值，在大规模真机示教与自主采集的数据上做离线 RL，并加了保守正则。
基准与经验。D4RL（Fu 等，2020）是离线 RL 最常用的基准，除了 MuJoCo 运动任务，还包括 Adroit 灵巧手和 Franka 厨房任务，数据集刻意分成"专家 / 中等 / 随机 / 回放"等不同质量。robomimic（Mandlekar 等，CoRL 2021）在机械臂操作任务上系统比较了 BC 与离线 RL：在人类示教数据上，带历史的 BC（BC-RNN）很难被超过；离线 RL 在质量参差的机器生成数据上更有优势。这和 FIG 5 的结论一致：数据越好、越"整齐"，BC 越够用。
离线到在线的真机强化学习。先用示教和旧数据预训练、再在真机上少量在线微调，是真机 RL 的主流路线之一。SERL（Luo 等，ICRA 2024）这类真机 RL 软件以 RLPD 式的"离线数据与在线数据混合采样"为核心，从少量示教出发在真实机械臂上学习电路板装配、线缆布线等操作任务（23.1 节）。
软件。d3rlpy（Seno 与 Imai）实现了 BCQ、CQL、IQL、TD3+BC 等算法；CORL（Tarasov 等）提供单文件的参考实现与基准结果；D4RL 的数据集现由 Farama 基金会的 Minari 维护。

什么时候用离线 RL？经验法则：数据里有奖励（或能事后标注），质量参差不齐，而且你需要比数据更好的行为；如果数据都是熟练的示教，先试 BC 及其变体（22.2 节的 ACT、Diffusion Policy）。另一个选择是条件化的序列模型，比如以目标回报为条件的 Decision Transformer（Chen 等，NeurIPS 2021），它把离线 RL 变成有条件的 BC。

Large-scale grasping: QT-Opt. Kalashnikov et al. (CoRL 2018) collected hundreds of thousands of grasp attempts on several real robot arms; most of the training data came from earlier policies (offline) and only part from online collection with the current policy. Q-learning with continuous actions (maximizing over actions with CEM, Lesson 6.11) learned closed-loop, vision-based grasping. It showed both the value of offline data and the difficulty of maximizing over actions.
Pre-training on multi-task robot data. PTR (Kumar et al., RSS 2023) pre-trains with CQL on a large multi-task robot dataset and then fine-tunes on a few demonstrations of a new task, needing far less data for the new task than training from scratch. Q-Transformer (Chebotar et al., CoRL 2023) discretizes each action dimension and outputs Q-values autoregressively one dimension at a time, doing offline RL on large sets of real demonstrations and autonomously collected data, with a conservative regularizer.
Benchmarks and lessons. D4RL (Fu et al., 2020) is the most used offline RL benchmark; besides MuJoCo locomotion it includes Adroit dexterous-hand and Franka kitchen tasks, with datasets deliberately split by quality ("expert / medium / random / replay" and so on). robomimic (Mandlekar et al., CoRL 2021) systematically compared BC and offline RL on arm manipulation: on human demonstrations, BC with history (BC-RNN) was hard to beat, while offline RL had more of an edge on machine-generated data of mixed quality. That agrees with FIG 5: the better and more "uniform" the data, the more BC suffices.
Offline-to-online RL on real robots. Pre-training on demonstrations and old data, then fine-tuning online for a short while on the robot, is one of the main routes for real-robot RL. Real-robot RL software such as SERL (Luo et al., ICRA 2024) is built around RLPD-style mixed sampling of offline and online data, learning manipulation tasks such as PCB assembly and cable routing on real arms from a small number of demonstrations (Lesson 23.1).
Software. d3rlpy (Seno & Imai) implements BCQ, CQL, IQL, TD3+BC and others; CORL (Tarasov et al.) provides single-file reference implementations with benchmark results; the D4RL datasets are now maintained in the Farama Foundation's Minari.

When to use offline RL? Rule of thumb: the data carry rewards (or can be labelled afterwards), are of mixed quality, and you need behaviour better than the data's; if the data are all skilled demonstrations, try BC and its variants first (ACT and Diffusion Policy in Lesson 22.2). Another option is a conditional sequence model such as the return-conditioned Decision Transformer (Chen et al., NeurIPS 2021), which turns offline RL into conditional BC.

机器人视角Robot's-eye view

离线 RL 的"分布外动作"在真机上有非常具体的含义：从没有人试过的高速、从没有人用过的抓取角度、从没有出现过的力。它们恰恰也是最可能损坏硬件的动作。所以离线 RL 的保守性不只是统计上的谨慎，也是安全上的谨慎；反过来，想超越数据，就要接受在线微调时付出的那部分真实代价，并用速度限幅、力矩限制和人工接管把它控制住。

On a real robot "out-of-distribution actions" mean something very concrete: speeds nobody has tried, grasp angles nobody has used, forces that never occurred. They are also exactly the actions most likely to damage hardware. So the conservatism of offline RL is not only statistical caution but safety caution; conversely, going beyond the data means accepting the real cost paid during online fine-tuning, and containing it with speed limits, torque limits and human takeover.

8常见误区Common pitfalls

误区 1：用学到的 Q 值来评价离线策略Pitfall 1: judging an offline policy by its own Q-values

学习器对自己的评估正是最不可信的东西：天真的方法会高估（FIG 2 的蓝线比绿线高出十几），保守的方法又会偏低。要评价离线策略，只能在环境里真跑（仿真或受控的真机测试），或者用专门的离策略评估（OPE）方法，而它们本身在分布外也不可靠。

The learner's own estimate is exactly what deserves the least trust: naive methods overestimate (FIG 2's blue line sits more than ten above the green one), conservative ones underestimate. An offline policy can only be judged by actually running it (in simulation or controlled real-robot tests) or by dedicated off-policy evaluation (OPE) methods, which are themselves unreliable outside the data.

误区 2：用在线测试调超参，却叫它"离线"Pitfall 2: tuning hyperparameters with online tests and calling it "offline"

CQL 的 $\alpha$、TD3+BC 的 $\alpha$、IQL 的 $\tau$ 和 $\beta$ 都很关键（FIG 3：同一种方法，旋钮换一下就从 8 步变成失败）。论文里的数字常常是在环境里试了很多组超参后选出来的。真正离线时没有这个机会，要么准备一个可用的评估途径，要么选对超参不敏感的方法和保守的默认值。

CQL's $\alpha$, TD3+BC's $\alpha$, and IQL's $\tau$ and $\beta$ all matter a lot (FIG 3: the same method goes from 8 steps to failure when a knob moves). Published numbers are often chosen after trying many hyperparameter settings in the environment. Truly offline you get no such chance: either arrange some way to evaluate, or pick methods that are insensitive to hyperparameters and conservative defaults.

误区 3：越保守越安全，所以越保守越好Pitfall 3: more conservative is safer, hence better

过度保守会把离线 RL 变成对最常见行为的模仿，而最常见的行为往往不是最好的：FIG 3 里 $\alpha=1$ 的 CQL 比 BC 还慢。保守的目的是不去利用未知，不是不去利用数据里已经证明好的动作。

Excessive conservatism turns offline RL into imitation of the most common behaviour, and the most common behaviour is often not the best: in FIG 3, CQL with $\alpha=1$ is slower than BC. The point of conservatism is to not exploit the unknown, not to avoid exploiting actions the data have already shown to be good.

误区 4：指望离线 RL 发现数据之外的好动作Pitfall 4: expecting offline RL to discover good actions beyond the data

离线 RL 能做的是在数据的支撑内重新组合（选好的动作、拼好的片段），不能凭空知道 0.75 比 0.55 更好：这需要有人试过 0.75。如果数据里关键的动作或状态完全没有覆盖，任何算法都无能为力（FIG 1 的覆盖任务），这时该做的是补数据或在线微调。

What offline RL can do is recombine within the support of the data (choose good actions, stitch good pieces); it cannot know out of thin air that 0.75 beats 0.55, because that needs someone to have tried 0.75. If crucial actions or states are not covered at all, no algorithm can help (FIG 1's coverage task); then the answer is more data or online fine-tuning.

误区 5：数据只要多就行，奖励和格式无所谓Pitfall 5: any large pile of data will do, rewards and formats don't matter

离线 RL 需要每条转移都有奖励，而且奖励要和你真正想要的东西一致；示教数据往往没有奖励，事后用规则或分类器标注时，标注的偏差会直接变成策略的偏差。还要注意终止标志（成功、失败和超时截断不同：超时不应当作终止）、状态与奖励的归一化，以及数据里的动作是否和部署时的控制频率、动作空间一致。

Offline RL needs a reward on every transition, and the reward must match what you actually want; demonstrations often have none, and when they are labelled afterwards by rules or a classifier, the labelling bias becomes the policy's bias. Also watch the termination flags (success, failure and time-limit truncation differ: a timeout should not be treated as terminal), normalization of states and rewards, and whether the logged actions match the control rate and action space used at deployment.

9小练习Check yourself

1. 在固定数据集上直接训练 DQN，Q 值常常越训越高、甚至高于任何可能的回报。最根本的原因是？1. Train a DQN directly on a fixed dataset and the Q-values often grow and grow, even above any possible return. What is the root cause?

选项 B。FIG 2 用了目标网络和很小的学习率，价值照样膨胀：问题在于数据外的 $\delta$ 没有约束，$\max$ 又专门挑偏高的那些，误差最多放大到 $\delta/(1-\gamma)$（第 6 节）。学习率和目标网络影响的是稳定性（选项 A、D），不能补上缺失的数据；奖励的大小只改变尺度（选项 C）。Option B. FIG 2 uses a target network and a small learning rate and the values inflate anyway: the error $\delta$ outside the data is unconstrained, $\max$ singles out the positive ones, and the error can grow to $\delta/(1-\gamma)$ (Section 6). Learning rate and target network affect stability (options A and D) but cannot supply missing data; the size of the rewards only changes the scale (option C).

2. 把 CQL 的 $\alpha$ 设得非常大，学到的贪心策略会趋向于什么？2. With a very large CQL $\alpha$, what does the learned greedy policy tend towards?

选项 C。只看正则项，平衡点是 $\operatorname{softmax}Q=\pi_\beta$，即 $Q=\log\pi_\beta+c$，$\arg\max$ 就是 $\pi_\beta$ 的众数，与回报无关。FIG 3 中 $\alpha=1$ 的 CQL 偏向谨慎操作员的慢速，比 BC（模仿的是均值）还慢。选项 D 是过滤式 BC 的思路。Option C. Looking at the regularizer alone, the balance point is $\operatorname{softmax}Q=\pi_\beta$, i.e. $Q=\log\pi_\beta+c$, whose $\arg\max$ is the mode of $\pi_\beta$, regardless of return. In FIG 3, CQL at $\alpha=1$ leans towards the cautious operator's slow speed and ends up slower than BC (which imitates the mean). Option D is the idea behind filtered BC.

3. IQL 中，对数据里某状态各动作的 $Q(s,a_i)$ 做 $\tau\to1$ 的期望分位回归，得到的 $V_\tau(s)$ 逼近什么？3. In IQL, expectile regression with $\tau\to1$ on the values $Q(s,a_i)$ of the logged actions at a state gives a $V_\tau(s)$ that approaches what?

选项 A。$V_\tau$ 是这些样本的一个加权平均（权重 $\tau$ 或 $1-\tau$），所以它总落在样本的最小值和最大值之间，$\tau\to1$ 时趋向最大值（FIG 4：$\tau=0.99$ 时 $-6.21$，最大值 $-6.00$）。它从不查询数据外的动作（排除 B）。均值是 $\tau=0.5$ 的情形（C）；中位数是分位数回归（绝对值损失）的 $\tau=0.5$，不是期望分位（D）。Option A. $V_\tau$ is a weighted average of the samples (weights $\tau$ or $1-\tau$), so it always lies between their minimum and maximum, and approaches the maximum as $\tau\to1$ (FIG 4: $-6.21$ at $\tau=0.99$, maximum $-6.00$). It never queries actions outside the data (not B). The mean is the $\tau=0.5$ case (C); the median is quantile regression (absolute loss) at 0.5, not an expectile (D).

4. 一份机器人数据里，每个回合都是熟练与笨拙动作的混合，没有哪个回合整体上特别好。哪种方法最有希望超过所有回合？4. In a robot dataset every episode mixes skilled and clumsy actions, and no episode is particularly good as a whole. Which method is most likely to beat all of them?

选项 D。好动作散落在各回合里，按回合过滤挑不出它们（B），BC 学的是平均（A）；离线 RL 用价值函数给每条转移打分，可以把好的片段拼起来（FIG 5"每步切换"：IQL 8 步，过滤式 BC 10 步，BC 12 步）。天真的离线 Q-learning 虽然也按转移判断，但会利用外推误差而失败（C，FIG 2）。Option D. The good actions are scattered across episodes, so filtering by episode cannot find them (B) and BC learns the average (A); offline RL scores every transition with a value function and can stitch the good pieces together (FIG 5, "switching every step": IQL 8 steps, filtered BC 10, BC 12). Naive offline Q-learning also judges transitions but fails by exploiting extrapolation error (C, FIG 2).

5. FIG 2 的在线模式能修好天真 Q-learning 的问题，原因是什么？5. Why does FIG 2's online mode fix naive Q-learning?

选项 B。算法完全一样，只是数据来源变了：策略想去哪里，数据就出现在哪里，于是 $\delta$ 在策略访问的地方又变成了可以被纠正的统计误差。代价是真实的失败（计数器上的倒箱数），这正是离线 RL 想避免、而在线微调要管控的东西。不是数据量本身（A），算法和 $\gamma$ 都没变（C、D）。Option B. The algorithm is identical; only the source of data changes: wherever the policy wants to go, data appear, so $\delta$ becomes a correctable statistical error again wherever the policy goes. The price is real failures (the tipped-box counter), exactly what offline RL tries to avoid and online fine-tuning must manage. It is not the amount of data as such (A), and neither the algorithm nor $\gamma$ changes (C, D).

10延伸阅读Further reading

S. Levine, A. Kumar, G. Tucker & J. Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, arXiv:2005.01643, 2020 — 最好的入门综述：分布偏移、策略约束、保守方法与开放问题the best entry-point review: distribution shift, policy constraints, conservative methods and open problems
D. Ernst, P. Geurts & L. Wehenkel, Tree-Based Batch Mode Reinforcement Learning, JMLR 2005; M. Riedmiller, Neural Fitted Q Iteration, ECML 2005 — 拟合 Q 迭代：batch RL 的经典形式fitted Q-iteration: the classic form of batch RL
S. Lange, T. Gabel & M. Riedmiller, Batch Reinforcement Learning, in Reinforcement Learning: State-of-the-Art, Springer, 2012 — 深度学习之前的 batch RL 综述a survey of batch RL before deep learning
S. Fujimoto, D. Meger & D. Precup, Off-Policy Deep Reinforcement Learning without Exploration, ICML 2019 — BCQ，并提出"外推误差"这个概念BCQ, and the notion of "extrapolation error"
A. Kumar, J. Fu, M. Soh, G. Tucker & S. Levine, Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction, NeurIPS 2019 — BEAR：用 MMD 约束策略的支撑BEAR: constraining the policy's support with an MMD
A. Kumar, A. Zhou, G. Tucker & S. Levine, Conservative Q-Learning for Offline Reinforcement Learning, NeurIPS 2020 — CQL 及其下界性质CQL and its lower-bound property
S. Fujimoto & S. S. Gu, A Minimalist Approach to Offline Reinforcement Learning, NeurIPS 2021 — TD3+BC：一行 BC 项的强基线TD3+BC: a strong baseline from one BC term
I. Kostrikov, A. Nair & S. Levine, Offline Reinforcement Learning with Implicit Q-Learning, ICLR 2022 — IQL：期望分位回归与优势加权的策略提取IQL: expectile regression and advantage-weighted policy extraction
X. B. Peng, A. Kumar, G. Zhang & S. Levine, Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning, arXiv:1910.00177, 2019; A. Nair, A. Gupta, M. Dalal & S. Levine, AWAC: Accelerating Online Reinforcement Learning with Offline Datasets, arXiv:2006.09359, 2020 — 加权回归式的策略提取与离线到在线weighted-regression policy extraction and offline-to-online learning
D. Brandfonbrener, W. F. Whitney, R. Ranganath & J. Bruna, Offline RL Without Off-Policy Evaluation, NeurIPS 2021 — "一步"方法为什么常常就够了why "one-step" methods are often enough
A. Kumar, J. Hong, A. Singh & S. Levine, When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?, ICLR 2022 — 离线 RL 与 BC 的比较（对应 FIG 5）offline RL versus BC (the question of FIG 5)
J. Fu, A. Kumar, O. Nachum, G. Tucker & S. Levine, D4RL: Datasets for Deep Data-Driven Reinforcement Learning, arXiv:2004.07219, 2020 — 最常用的离线 RL 基准the most widely used offline RL benchmark
A. Mandlekar et al., What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, CoRL 2021 — robomimic：机械臂操作上 BC 与离线 RL 的系统比较robomimic: a systematic comparison of BC and offline RL on arm manipulation
D. Kalashnikov et al., QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation, CoRL 2018; Y. Chebotar et al., Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, CoRL 2023 — 大规模真机数据上的（离线）Q-learning(offline) Q-learning on large real-robot datasets
P. J. Ball, L. Smith, I. Kostrikov & S. Levine, Efficient Online Reinforcement Learning with Offline Data, ICML 2023; M. Nakamoto et al., Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning, NeurIPS 2023 — RLPD 与 Cal-QL：从离线到在线RLPD and Cal-QL: from offline to online