逆强化学习与偏好学习Inverse RL & Preference Learning

20.1 节到 20.5 节的所有算法都从一个奖励函数 \(r(s,a)\) 出发：值迭代、Q-learning、PPO、SAC 都只是在最大化它的期望回报。可是在真实任务里，奖励不是环境给的，而是工程师写的。一台越野小车该怎样在“走草地”“绕远路”“冒险穿过泥坑”之间权衡？一只机械臂把杯子递给人时，“动作自然”怎么写成公式？手写的奖励常常漏掉某个条件，强化学习又极其擅长钻这种空子：它会找到一种让数字变大、却完全不是你想要的行为（reward hacking）。

另一方面，人往往做得出来或者评判得出来，只是说不清楚。司机会开，却说不出自己给泥坑的权重是多少；你分不清两段抓取动作各自得几分，却能马上说出哪一段更好。于是有两条路：

• 逆强化学习（IRL）：给定专家的示教轨迹，反推一个奖励，使专家的行为在这个奖励下是（近似）最优的；然后用这个奖励在新环境里重新规划或训练策略。GAIL 是它的一个近亲：不显式输出奖励，而是用一个判别器（19.2 节）当奖励直接训练策略。
• 偏好学习：让人比较两段行为“哪个更好”，用 Bradley–Terry 模型拟合一个奖励模型，再用强化学习（RLHF）或直接用偏好对（DPO）微调策略。今天的大语言模型就是这样对齐的，同样的工具也正在用于机器人策略。

和行为克隆（22.1 节）相比，学奖励多绕了一步，换来的是“为什么”：奖励描述的是目标，而不是在某个地方该往哪走，所以地图变了、障碍物挪了，机器人可以用同一个奖励重新规划（FIG 4）。

Every algorithm from Lesson 20.1 to Lesson 20.5 starts from a reward function \(r(s,a)\): value iteration, Q-learning, PPO and SAC all just maximize its expected return. In a real task, however, the reward is not handed over by the environment; an engineer writes it. How should an off-road robot trade "drive on grass" against "take the long way" against "risk crossing the mud"? When an arm hands a cup to a person, how do you write "moves naturally" as a formula? Hand-written rewards often miss a condition, and reinforcement learning is extremely good at exploiting exactly that: it finds a behaviour that makes the number large and is not at all what you wanted (reward hacking).

On the other hand, people can often do the task or judge it while being unable to state it. A driver can drive but cannot tell you what weight they give to mud; you cannot score two grasping motions on an absolute scale, but you can say at once which one is better. That gives two routes:

• Inverse reinforcement learning (IRL): given expert demonstrations, infer a reward under which the expert's behaviour is (nearly) optimal, then use that reward to re-plan or train a policy in new situations. GAIL is a close relative: it never outputs a reward explicitly, but trains the policy directly with a discriminator (Lesson 19.2) as the reward.
• Preference learning: ask a person which of two behaviours is better, fit a reward model with the Bradley–Terry model, then fine-tune the policy with reinforcement learning (RLHF) or directly from the preference pairs (DPO). This is how today's large language models are aligned, and the same tools are being applied to robot policies.

Compared with behaviour cloning (Lesson 22.1), learning a reward takes a detour and buys the "why": a reward describes the goal rather than which way to move in a given place, so when the map changes or an obstacle moves, the robot can re-plan with the same reward (FIG 4).

正向的强化学习是“给定奖励，求最优行为”；逆强化学习是“给定行为，求奖励”。逆问题的通病是解不唯一。本节贯穿全篇的例子是一台越野小车：在 10×7 的格子地图上从左边的 S 开到右边的 G，每一步走到上下左右相邻的格子。地面有三种：铺装路面、草地、泥地。我们假设奖励是地形特征的线性组合，每走进一个格子 \(s'\) 得到

Forward reinforcement learning is "given a reward, find the optimal behaviour"; inverse reinforcement learning is "given behaviour, find the reward". Inverse problems share a common disease: the answer is not unique. The running example of this lesson is an off-road robot on a 10×7 grid map, driving from S on the left to G on the right, one step at a time to a neighbouring cell. There are three kinds of ground: pavement, grass and mud. We assume the reward is linear in terrain features: every move into a cell \(s'\) earns

其中 \(\phi_{\mathrm g}(s')\) 在 \(s'\) 是草地时为 1、否则为 0，\(\phi_{\mathrm m}\) 对泥地同理，\(w_{\mathrm s}\) 是每走一步的奖励（负数就是时间代价）。\(\mathbf f(\tau)\) 是轨迹的特征计数：走了几步、踩了几格草、几格泥。到达 G 就结束（回报不打折，\(\gamma=1\)）。下图左边紫色是专家的路线：先往上一格，沿着草带绕过泥地，共 11 步、踩了 8 格草。右边是奖励权重的平面（横轴 \(w_{\mathrm g}\)，纵轴 \(w_{\mathrm m}\)，\(w_{\mathrm s}\) 固定为 −1），拖动蓝点改变奖励，左图的蓝色路线就是这个奖励下的最优路线（6.12 节的值迭代）。青色区域是所有“让专家路线成为最优”的奖励，也就是 IRL 的全部候选答案。

Here \(\phi_{\mathrm g}(s')\) is 1 if \(s'\) is grass and 0 otherwise, \(\phi_{\mathrm m}\) likewise for mud, and \(w_{\mathrm s}\) is the reward for each step (negative: a cost of time). \(\mathbf f(\tau)\) are the trajectory's feature counts: how many steps, how many grass cells, how many mud cells. Reaching G ends the episode (no discounting, \(\gamma=1\)). In the figure, purple on the left is the expert's route: one cell up, then along the grass band around the mud, 11 steps with 8 grass cells. On the right is the plane of reward weights (\(w_{\mathrm g}\) across, \(w_{\mathrm m}\) up, \(w_{\mathrm s}\) fixed at −1). Drag the blue point to change the reward; the blue route on the left is the optimal route under it (value iteration from Lesson 6.12). The cyan region is every reward that makes the expert's route optimal: all of IRL's candidate answers.

青色区域的边界可以直接从三条路线的回报读出来（\(w_{\mathrm s}=-1\)）：专家的草带路线回报是 \(-11+8w_{\mathrm g}\)，直穿泥地是 \(-9+2w_{\mathrm g}+4w_{\mathrm m}\)，最上面的铺装绕行是 \(-13\)。专家路线最优，当且仅当 \(w_{\mathrm g}>-0.25\)（否则铺装绕行更好）且 \(w_{\mathrm m}<-0.5+1.5\,w_{\mathrm g}\)（否则直穿更好）。这是一个无界的楔形：只要泥“足够糟”，再糟多少都一样；草甚至可以是加分的。

The edges of the cyan region can be read off the returns of three routes (with \(w_{\mathrm s}=-1\)): the expert's grass-band route returns \(-11+8w_{\mathrm g}\), straight through the mud \(-9+2w_{\mathrm g}+4w_{\mathrm m}\), and the pavement detour along the top \(-13\). The expert's route is optimal exactly when \(w_{\mathrm g}>-0.25\) (otherwise the pavement detour wins) and \(w_{\mathrm m}<-0.5+1.5\,w_{\mathrm g}\) (otherwise going straight wins). It is an unbounded wedge: once mud is "bad enough", any amount worse is the same; grass may even be a bonus.

真实的示教不是一条完美路线，而是很多条有点随意的路线：有人多拐了一下，有人偶尔抄近路。Ziebart 等（2008）的最大熵 IRL 把这种随意性写进模型：专家按回报的指数挑路线，回报越高越常走，但每条路线都有可能，

Real demonstrations are not one perfect route but many somewhat casual ones: one person takes an extra turn, another cuts a corner. The maximum-entropy IRL of Ziebart et al. (2008) builds that casualness into the model: the expert picks routes with probability exponential in their return, so better routes are more common but every route is possible,

这是一个指数族分布（19.5 节的能量模型，能量就是 \(-R_{\mathbf w}\)），它的对数似然对 \(\mathbf w\) 是凹的，梯度只有一句话：示教的平均特征计数减去模型的期望特征计数。梯度为零时两者相等，这叫特征匹配：学到的模型平均走的步数、踩的草、踩的泥和示教一模一样。期望特征计数不用枚举所有路线，用两遍动态规划就能精确算出：一遍向后的“软”值迭代（max 换成 log-sum-exp），得到每一步的随机策略；一遍向前传播状态分布，数出每种特征的期望次数（第 7 节）。

下图用 \(\mathbf w^{\star}=(-2,-0.3,-4)\)（步数、草、泥）按上式随机生成 \(N\) 条示教，然后从 \(\mathbf w=\mathbf 0\)（所有在 24 步内到得了 G 的路线等可能）开始梯度上升。注意这里 \(w_{\mathrm s}\) 也要学：最大熵模型里奖励的大小有意义，它决定专家有多“随意”。

This is an exponential-family distribution (an energy-based model as in Lesson 19.5, with energy \(-R_{\mathbf w}\)); its log-likelihood is concave in \(\mathbf w\), and the gradient is one sentence: the demonstrations' mean feature counts minus the model's expected feature counts. At zero gradient they are equal, which is called feature matching: on average the learned model takes as many steps, crosses as much grass and as much mud as the demonstrations. The expected counts need no enumeration of routes; two dynamic-programming sweeps give them exactly: a backward "soft" value iteration (max replaced by log-sum-exp) yields a stochastic policy for every time step, and a forward pass of the state distribution counts the expected number of each feature (Section 7).

The figure draws \(N\) demonstrations at random from the formula above with \(\mathbf w^{\star}=(-2,-0.3,-4)\) (step, grass, mud), then runs gradient ascent from \(\mathbf w=\mathbf 0\) (every route that reaches G within 24 steps equally likely). Note that \(w_{\mathrm s}\) is learned as well: in the maximum-entropy model the size of the reward matters, because it sets how "casual" the expert is.

最大熵 IRL 的内循环要反复“在当前奖励下求最优策略”，这在格子世界里是两遍动态规划，在一台连续控制的机器人上就是一整轮强化学习。Ho 与 Ermon（2016）注意到，如果最终要的只是一个会模仿的策略，奖励只是中间产物，那么“IRL 之后再 RL”可以合并成一个问题：让策略的状态-动作占用度量 \(\rho_\pi(s,a)\) 和专家的 \(\rho_E(s,a)\) 一样。度量两个分布的差别，正是 19.2 节判别器的本事：

The inner loop of maximum-entropy IRL repeatedly "solves for the optimal policy under the current reward": two dynamic-programming sweeps on a grid, a full round of reinforcement learning on a robot with continuous controls. Ho and Ermon (2016) observed that if all you want in the end is a policy that imitates, the reward is only an intermediate product, and "IRL, then RL" can be merged into one problem: make the policy's state-action occupancy measure \(\rho_\pi(s,a)\) equal to the expert's \(\rho_E(s,a)\). Measuring the difference between two distributions is exactly what the discriminator of Lesson 19.2 does:

判别器 \(D(s,a)\) 学“这个状态-动作对来自专家的概率”；策略用强化学习最大化一个由 \(D\) 给出的奖励，做得越像专家，得分越高。这里写的是 FIG 3 用的 \(\log D\)；原论文用的是 \(-\log(1-D)\)，两者的平衡点相同，区别见第 7 节。它就是 19.2 节的 GAN，只不过“生成器”是一个在环境里行动的策略，样本要靠执行策略得到，梯度不能穿过环境传回来，所以要用强化学习（原文用 TRPO）而不是反向传播来更新策略。

下图在同一张地图上运行 GAIL。为了让图里没有采样噪声，占用度量是精确算出来的（真实的 GAIL 从执行策略得到的样本里估计），每次迭代：判别器（2→24→24→1 的 MLP，输入是格子坐标和移动方向，看不到地形，就像真实的 GAIL 只看原始状态）做 2 步 Adam；策略向当前奖励 \(\log D\) 下的熵正则最优策略（温度 \(\lambda=0.3\) 的软值迭代）迈半步。

The discriminator \(D(s,a)\) learns "the probability that this state-action pair came from the expert"; the policy maximizes a reward given by \(D\) with reinforcement learning: the more expert-like, the higher the score. Written here is the \(\log D\) used in FIG 3; the original paper uses \(-\log(1-D)\), with the same equilibrium; Section 7 explains the difference. It is the GAN of Lesson 19.2, except that the "generator" is a policy acting in an environment. Its samples come from running the policy and gradients cannot flow back through the environment, so the policy is updated by reinforcement learning (TRPO in the original paper), not by backpropagation.

The figure runs GAIL on the same map. To keep sampling noise out of the picture, the occupancy measures are computed exactly (real GAIL estimates them from rollouts). Each iteration: the discriminator (a 2→24→24→1 MLP whose input is the cell coordinates and the move direction; it does not see the terrain, just as real GAIL sees only raw states) takes 2 Adam steps; the policy takes half a step towards the entropy-regularized optimal policy under the current reward \(\log D\) (soft value iteration at temperature \(\lambda=0.3\)).

在训练用的地图上，行为克隆、GAIL 和 IRL 给出的路线一样好。区别出现在新环境里。下图把三样东西都搬到一张新地图上：行为克隆（每个格子取示教里最常见的移动，没去过的格子照抄最近的去过的格子）、FIG 3 那样训练 300 次迭代的 GAIL 策略、以及用 FIG 2 那样学到的奖励 \(\mathbf w=(-1.98,-0.23,-2.92)\)（20 条示教、400 次迭代）在新地图上重新规划的路线。三者用的是同一份 20 条示教。

On the training map, behaviour cloning, GAIL and IRL give equally good routes. The difference shows up in a new environment. The figure moves all three to a new map: behaviour cloning (in each cell, the most common move in the demonstrations; in cells never visited, the move of the nearest visited cell), a GAIL policy trained for 300 iterations as in FIG 3, and the route re-planned on the new map with a reward learned as in FIG 2, \(\mathbf w=(-1.98,-0.23,-2.92)\) (20 demonstrations, 400 iterations). All three use the same 20 demonstrations.

示教要求人会做，偏好只要求人会比。Bradley–Terry 模型（1952）假设每段行为有一个潜在的分数 \(R\)，人选 A 而不选 B 的概率是分数差的 sigmoid：

Demonstrations require a person who can do the task; preferences only require a person who can compare. The Bradley–Terry model (1952) assumes each behaviour has a latent score \(R\), and the probability that a person picks A over B is the sigmoid of the score difference:

拟合 \(\psi\) 就是在“分数差”上做逻辑回归（16.1 节）。它也是 FIG 2 的最大熵模型在“只有两个候选”时的特例：在 A、B 两条轨迹之间按 \(e^{R}\) 挑一条。下图里一台移动机器人要从左到右绕过一个障碍物，候选路线有 80 条，各自的偏移和抖动不同。每条路线有三个特征：比直线多走的长度 \(\phi_1\)、靠近障碍物的风险 \(\phi_2\)（最小间隙越小越接近 1，撞上时为 1）、抖动程度 \(\phi_3\)。一个“模拟的人”心里的分数是 \(R^*=-5\phi_1-8\phi_2-5\phi_3\)，按 \(P(A\succ B)=\sigma(\beta(R^*_A-R^*_B))\) 回答，\(\beta\) 越大这个人越一致。奖励模型 \(R_\psi=\psi^\top\boldsymbol\phi\) 对三个特征线性，每来一个标注就用牛顿法重新拟合（带一个很小的 L2 正则 \(0.01\|\psi\|^2\)，避免数据完全可分时 \(\psi\) 无界）。你也可以自己当标注员。

Fitting \(\psi\) is logistic regression on score differences (Lesson 16.1). It is also FIG 2's maximum-entropy model in the special case of only two candidates: choose between trajectories A and B with probability proportional to \(e^{R}\). In the figure a mobile robot must get from left to right around an obstacle, and there are 80 candidate paths with different offsets and wiggles. Each path has three features: the extra length beyond a straight line \(\phi_1\), the risk of passing close to the obstacle \(\phi_2\) (closer to 1 as the minimum clearance shrinks, 1 on contact), and how much it wiggles \(\phi_3\). A "simulated person" scores paths with \(R^*=-5\phi_1-8\phi_2-5\phi_3\) and answers with \(P(A\succ B)=\sigma(\beta(R^*_A-R^*_B))\); the larger \(\beta\), the more consistent the person. The reward model \(R_\psi=\psi^\top\boldsymbol\phi\) is linear in the three features and is re-fitted by Newton's method after every label (with a tiny L2 penalty \(0.01\|\psi\|^2\) so that \(\psi\) stays bounded when the data are perfectly separable). You can also be the labeller yourself.

注意 BT 模型只看分数差：给所有路线的分数加同一个常数，似然不变，所以奖励模型的零点永远无法确定（FIG 1 的歧义在这里又出现了一次）。缩放也和标注员的一致性 \(\beta\) 纠缠在一起：一致的人（\(\beta\) 大）让 \(\psi\) 的绝对值变大，随意的人让它变小。能可靠学到的只有方向，也就是各特征之间的相对权重。

Note that the BT model sees only score differences: add the same constant to every path's score and the likelihood does not change, so the zero point of a reward model can never be determined (FIG 1's ambiguity once again). The scale is entangled with the labeller's consistency \(\beta\): a consistent person (large \(\beta\)) makes \(|\psi|\) large, a careless one makes it small. What can be learned reliably is the direction, the relative weights of the features.

有了奖励模型 \(\hat r\)，就可以优化策略。但如果直接最大化 \(\hat r\)，策略会跑到奖励模型从没见过、因而估得不准的地方去。Christiano 等（2017）最早把“从比较里学奖励模型、再用深度 RL 优化它”做成了可行的流程；用于语言模型时，标准做法（Ziegler 等，2019；Ouyang 等，2022）是再加一个 KL 约束，让策略不要离一个参考策略 \(\pi_{\mathrm{ref}}\)（通常是监督微调或行为克隆得到的策略）太远：

With a reward model \(\hat r\) in hand we can optimize a policy. Maximize \(\hat r\) directly, however, and the policy runs off to places the reward model has never seen and therefore estimates badly. Christiano et al. (2017) made "learn a reward model from comparisons, then optimize it with deep RL" practical; for language models the standard recipe (Ziegler et al., 2019; Ouyang et al., 2022) adds a KL constraint that keeps the policy close to a reference policy \(\pi_{\mathrm{ref}}\) (usually the result of supervised fine-tuning or behaviour cloning):

右边的闭式解（第 7 节推导）说明 KL 正则的最优策略就是把参考策略按 \(e^{\hat r/\beta}\) 重新加权。在语言模型或机器人策略上 \(a\) 的空间太大，没法归一化，所以 RLHF 用 PPO（20.4 节）去逼近它。DPO（Rafailov 等，2023）反过来用：既然 \(\hat r(a)=\beta\log\frac{\pi^*(a)}{\pi_{\mathrm{ref}}(a)}+\beta\log Z\)，就把这个式子代进 BT 损失，\(Z\) 在分数差里消掉，直接在偏好对上训练策略，不需要单独的奖励模型，也不需要强化学习：

The closed form on the right (derived in Section 7) says that the KL-regularized optimal policy is just the reference policy re-weighted by \(e^{\hat r/\beta}\). For a language model or a robot policy the space of \(a\) is far too large to normalize, so RLHF approximates it with PPO (Lesson 20.4). DPO (Rafailov et al., 2023) runs the formula backwards: since \(\hat r(a)=\beta\log\frac{\pi^*(a)}{\pi_{\mathrm{ref}}(a)}+\beta\log Z\), substitute it into the BT loss, where \(Z\) cancels in the score difference, and train the policy directly on the preference pairs, with no separate reward model and no reinforcement learning:

下图是一个最小的例子：同一个杯子有 6 个候选抓取，参考策略（从示教克隆来的）偏爱前两个，而真正成功率最高的是抓取 4。偏好数据是从参考策略里成对抽取抓取，再按 BT 模型和真实奖励 \(r^*\) 随机标注“哪个更好”。RLHF 一侧先拟合表格形式的奖励模型 \(\hat r\)，再用上面的闭式解得到 \(\pi^*\)（相当于 PPO 完美收敛）；DPO 一侧从 \(\pi_{\mathrm{ref}}\) 出发，用 Adam 直接最小化 \(\mathcal L_{\mathrm{DPO}}\)。两边用同一份数据、同一个极小的 L2 正则（\(0.001\sum_a(r(a)-\bar r)^2\)，DPO 加在它隐含的奖励 \(\beta\log\frac{\pi_\theta}{\pi_{\mathrm{ref}}}\) 上）。

The figure is a minimal example: one mug, 6 candidate grasps; the reference policy (cloned from demonstrations) favours the first two, while the grasp that actually succeeds most often is grasp 4. Preference data are made by drawing pairs of grasps from the reference policy and labelling "which is better" at random with the BT model and the true reward \(r^*\). The RLHF side first fits a tabular reward model \(\hat r\), then takes \(\pi^*\) from the closed form above (as if PPO had converged perfectly); the DPO side starts from \(\pi_{\mathrm{ref}}\) and minimizes \(\mathcal L_{\mathrm{DPO}}\) directly with Adam. Both use the same data and the same tiny L2 penalty (\(0.001\sum_a(r(a)-\bar r)^2\), applied in DPO to its implied reward \(\beta\log\frac{\pi_\theta}{\pi_{\mathrm{ref}}}\)).

MDP \((\mathcal S,\mathcal A,P,\gamma)\) 已知，奖励 \(r\) 未知；给定专家策略 \(\pi_E\)（或它的示教）。IRL 要找 \(r\)，使 \(\pi_E\) 在 \(r\) 下最优，即对每个状态，专家的动作的 \(Q\) 值不低于任何别的动作（20.1 节的 Bellman 最优方程）：

The MDP \((\mathcal S,\mathcal A,P,\gamma)\) is known, the reward \(r\) is not; we are given an expert policy \(\pi_E\) (or demonstrations of it). IRL seeks \(r\) such that \(\pi_E\) is optimal under \(r\): in every state the expert's action has a \(Q\)-value at least as high as any other action (the Bellman optimality equation of Lesson 20.1):

对有限 MDP，\(Q^{\pi_E}_r\) 对 \(r\) 是线性的（\(V^{\pi}=(I-\gamma P_\pi)^{-1}r_\pi\)），所以 \(\mathcal R(\pi_E)\) 是由线性不等式围成的凸锥：它包含 \(r=0\)，对正数缩放封闭，这就是 FIG 1 的青色楔形。Ng 与 Russell（2000）对状态奖励写出了这组不等式，并用线性规划在其中挑一个“让专家动作领先得最多”的奖励（最大间隔），再加 L1 正则偏好简单的奖励。最大间隔规划（Ratliff、Bagnell 与 Zinkevich，2006）把同样的思想推广到结构化输出：专家路径的回报要比其他每条路径高出一个与“差多少”成比例的间隔。

For a finite MDP, \(Q^{\pi_E}_r\) is linear in \(r\) (\(V^{\pi}=(I-\gamma P_\pi)^{-1}r_\pi\)), so \(\mathcal R(\pi_E)\) is a convex cone cut out by linear inequalities: it contains \(r=0\) and is closed under positive scaling, and it is FIG 1's cyan wedge. Ng and Russell (2000) wrote these inequalities for state rewards and used a linear program to pick, among them, the reward under which the expert's actions lead by the largest margin, with an L1 penalty favouring simple rewards. Maximum margin planning (Ratliff, Bagnell and Zinkevich, 2006) extends the idea to structured outputs: the expert's path must beat every other path by a margin proportional to how different that path is.

对任何函数 \(\Phi:\mathcal S\to\mathbb R\)，令 \(r'(s,a,s')=r(s,a,s')+\hl{shape}{\gamma\Phi(s')-\Phi(s)}\)。沿一条轨迹求和，中间项两两抵消：

For any function \(\Phi:\mathcal S\to\mathbb R\), let \(r'(s,a,s')=r(s,a,s')+\hl{shape}{\gamma\Phi(s')-\Phi(s)}\). Summed along a trajectory, the middle terms cancel in pairs:

每个状态的所有动作的 \(Q\) 值减去同一个数，\(\arg\max\) 不变（Ng、Harada 与 Russell，1999；他们还证明了：在不知道转移模型的前提下，势函数形式是唯一一种保证不改变最优策略的整形）。FIG 1 用 \(\gamma=1\)、\(\Phi(s)=-k\,d(s)\)，\(d\) 是到 G 的曼哈顿距离，于是每条从 S 到 G 的路线的回报都恰好加了 \(\Phi(G)-\Phi(S)=9k\)。所以 IRL 从示教里最多只能把奖励确定到“差一个整形项”。Skalse 等（2023）系统地刻画了不同数据源（最优示教、玻尔兹曼示教、偏好）能确定奖励到什么程度。

Every action's \(Q\)-value in a state drops by the same number, so the \(\arg\max\) does not change (Ng, Harada and Russell, 1999; they also showed that, without knowledge of the transition model, the potential form is the only shaping guaranteed to leave optimal policies unchanged). FIG 1 uses \(\gamma=1\) and \(\Phi(s)=-k\,d(s)\), with \(d\) the Manhattan distance to G, so every route from S to G gains exactly \(\Phi(G)-\Phi(S)=9k\). IRL can therefore pin a reward down at best "up to a shaping term". Skalse et al. (2023) characterize systematically how far different data sources (optimal demonstrations, Boltzmann demonstrations, preferences) determine a reward.

原则。在所有满足特征匹配 \(\mathbb E_p[\mathbf f]=\bar{\mathbf f}_{\mathcal D}\) 的路线分布 \(p\) 里，选熵最大的那个（4.11 节的熵：除了数据要求的，不多假设任何东西）。写拉格朗日函数 \(\mathcal L=-\sum_\tau p\log p+\mathbf w^\top(\sum_\tau p\,\mathbf f-\bar{\mathbf f}_{\mathcal D})+\mu(\sum_\tau p-1)\)，对 \(p(\tau)\) 求导置零，得 \(\log p(\tau)=\mathbf w^\top\mathbf f(\tau)+\mu-1\)，也就是第 3 节的 \(p_{\mathbf w}\propto e^{\mathbf w^\top\mathbf f}\)：拉格朗日乘子 \(\mathbf w\) 就是奖励权重。它的对偶问题恰好是最大化示教的对数似然，

Principle. Among all route distributions \(p\) that match the features, \(\mathbb E_p[\mathbf f]=\bar{\mathbf f}_{\mathcal D}\), pick the one with maximum entropy (the entropy of Lesson 4.11: assume nothing beyond what the data require). The Lagrangian is \(\mathcal L=-\sum_\tau p\log p+\mathbf w^\top(\sum_\tau p\,\mathbf f-\bar{\mathbf f}_{\mathcal D})+\mu(\sum_\tau p-1)\); setting its derivative with respect to \(p(\tau)\) to zero gives \(\log p(\tau)=\mathbf w^\top\mathbf f(\tau)+\mu-1\), which is the \(p_{\mathbf w}\propto e^{\mathbf w^\top\mathbf f}\) of Section 3: the Lagrange multipliers \(\mathbf w\) are the reward weights. Its dual problem is exactly maximizing the log-likelihood of the demonstrations,

Hessian 是负的特征协方差，所以 \(\ell\) 是凹的，梯度上升不会卡在局部最优。协方差在某个方向上为零（例如示教和模型都几乎不踩泥）时，\(\ell\) 在那个方向上几乎是平的，这就是 FIG 2 里泥的权重一直漂的原因。

计算 \(\mathbb E_{p_{\mathbf w}}[\mathbf f]\)。对确定性转移，“按 \(e^{R}\) 挑动作序列”可以分解成一个随时间变化的策略。FIG 2 和 FIG 3 用长度 \(T=24\) 的视界，并要求在最后一步时位于 G（G 是吸收状态，留在 G 不再得奖励）：

The Hessian is minus the feature covariance, so \(\ell\) is concave and gradient ascent cannot get stuck in a local optimum. Where the covariance vanishes in some direction (say, neither the demonstrations nor the model ever touch mud), \(\ell\) is almost flat along it, which is why the mud weight in FIG 2 keeps drifting.

Computing \(\mathbb E_{p_{\mathbf w}}[\mathbf f]\). With deterministic transitions, "choose the action sequence with probability \(\propto e^{R}\)" factorizes into a time-varying policy. FIGs 2 and 3 use a horizon of \(T=24\) and require being at G at the last step (G is absorbing, and staying there earns nothing):

向后这一遍就是 6.12 节的值迭代把 \(\max\) 换成了 \(\log\sum\exp\)（“软”最大值），\(V_0(S)=\log Z(\mathbf w)\)；向前这一遍是把起点的概率质量沿策略推开。两遍的代价都是 \(O(T\,|\mathcal S|\,|\mathcal A|)\)，FIG 2 每次迭代不到 1 ms。对随机转移，直接按 \(e^{R}\) 给轨迹加权会让智能体“相信自己能挑运气”；Ziebart 等（2010）的最大因果熵改为只对策略的因果熵最大化，结果是同样的软 Bellman 递推，只是 \(V_{t+1}(s')\) 换成 \(\mathbb E_{s'}[V_{t+1}(s')]\)。这也正是 SAC（20.4 节）里的软 Bellman 方程：最大熵 IRL 和最大熵 RL 是同一个模型的逆问题与正问题。

深度版本。把 \(\mathbf w^\top\boldsymbol\phi\) 换成神经网络 \(r_\theta(s)\)（Wulfmeier 等，2015），梯度变成 \(\sum_s(\bar d_{\mathcal D}(s)-d_{\theta}(s))\nabla_\theta r_\theta(s)\)：示教访问多于模型的状态奖励上调，反之下调。在连续空间里没法精确算 \(d_\theta\)，引导代价学习（Finn、Levine 与 Abbeel，2016）用当前策略的样本做重要性采样来估计 \(\log Z\)，策略同时用 RL 更新，这已经和下面的 GAIL 很像了。

The backward sweep is the value iteration of Lesson 6.12 with \(\max\) replaced by \(\log\sum\exp\) (a "soft" maximum), and \(V_0(S)=\log Z(\mathbf w)\); the forward sweep pushes the probability mass at the start out along the policy. Both cost \(O(T\,|\mathcal S|\,|\mathcal A|)\); one iteration of FIG 2 takes under 1 ms. With stochastic transitions, weighting trajectories directly by \(e^{R}\) makes the agent "believe it can choose its luck"; the maximum causal entropy formulation of Ziebart et al. (2010) maximizes only the causal entropy of the policy instead, and the result is the same soft Bellman recursion with \(V_{t+1}(s')\) replaced by \(\mathbb E_{s'}[V_{t+1}(s')]\). That is exactly the soft Bellman equation of SAC (Lesson 20.4): maximum-entropy IRL and maximum-entropy RL are the inverse and forward problems of the same model.

Deep versions. Replace \(\mathbf w^\top\boldsymbol\phi\) by a neural network \(r_\theta(s)\) (Wulfmeier et al., 2015) and the gradient becomes \(\sum_s(\bar d_{\mathcal D}(s)-d_{\theta}(s))\nabla_\theta r_\theta(s)\): raise the reward of states the demonstrations visit more than the model, lower it where the model visits more. In continuous spaces \(d_\theta\) cannot be computed exactly; guided cost learning (Finn, Levine and Abbeel, 2016) estimates \(\log Z\) by importance sampling from the current policy, which is itself updated with RL, and that is already very close to GAIL below.

策略的占用度量 \(\rho_\pi(s,a)=\sum_t\gamma^t P(s_t=s,a_t=a\mid\pi)\) 和策略一一对应（\(\pi(a|s)=\rho_\pi(s,a)/\sum_{a'}\rho_\pi(s,a')\)），任何奖励的期望回报都是它的线性函数 \(\sum_{s,a}\rho_\pi(s,a)r(s,a)\)。Ho 与 Ermon（2016）证明：用凸正则 \(\psi(r)\) 的最大因果熵 IRL 再接 RL，所得策略等于

A policy's occupancy measure \(\rho_\pi(s,a)=\sum_t\gamma^t P(s_t=s,a_t=a\mid\pi)\) corresponds one-to-one with the policy (\(\pi(a|s)=\rho_\pi(s,a)/\sum_{a'}\rho_\pi(s,a')\)), and every reward's expected return is linear in it, \(\sum_{s,a}\rho_\pi(s,a)r(s,a)\). Ho and Ermon (2016) proved that maximum-causal-entropy IRL with a convex regularizer \(\psi(r)\), followed by RL, yields the policy

其中 \(\psi^*\) 是 \(\psi\) 的凸共轭，它就是度量“两个占用度量差多少”的距离。选一个特定的 \(\psi\)，\(\psi^*\) 就变成（差一个常数的）\(\max_D\mathbb E_{\rho_\pi}[\log(1-D)]+\mathbb E_{\rho_E}[\log D]\)，也就是 19.2 节的 GAN 目标与 JS 散度；选“奖励在某个线性特征类里”，就退化成特征匹配。实际算法交替：判别器在专家样本和策略样本上做二分类，策略用 TRPO/PPO 最大化判别器给的奖励。

奖励的符号与偏置。原文的判别器输出“来自策略”的概率，策略的代价是 \(\log D_\pi\)，换成本页的写法就是奖励 \(-\log(1-D)\ge0\)。另一种常见选择是本页 FIG 3 用的 \(\log D\le0\)，还有 AIRL 风格的 \(\log D-\log(1-D)\)。三者的不动点相同，但在会提前结束的任务里偏置不同：恒正的奖励鼓励“活得越久越好”，恒负的奖励鼓励尽快结束（Kostrikov 等，2019 详细分析了这一点，并给出处理吸收状态的办法）。FIG 3 要求在 24 步内到达 G，选 \(\log D\) 时策略不会故意绕圈子。

where \(\psi^*\) is the convex conjugate of \(\psi\), and it is the distance that measures "how different two occupancy measures are". For one particular \(\psi\), \(\psi^*\) becomes (up to a constant) \(\max_D\mathbb E_{\rho_\pi}[\log(1-D)]+\mathbb E_{\rho_E}[\log D]\), the GAN objective and JS divergence of Lesson 19.2; for "rewards in a linear feature class" it reduces to feature matching. The practical algorithm alternates: the discriminator classifies expert samples against policy samples, and the policy maximizes the discriminator's reward with TRPO/PPO.

The sign of the reward and its bias. In the original paper the discriminator outputs the probability of "came from the policy" and the policy's cost is \(\log D_\pi\), which in this page's notation is the reward \(-\log(1-D)\ge0\). Another common choice is the \(\log D\le0\) used in FIG 3, and there is the AIRL-style \(\log D-\log(1-D)\). All three have the same fixed point but different biases in tasks that can end early: an always-positive reward encourages "survive as long as possible", an always-negative one encourages ending quickly (Kostrikov et al., 2019 analyse this in detail and show how to handle absorbing states). FIG 3 requires reaching G within 24 steps, and with \(\log D\) the policy has no incentive to circle around on purpose.

记 \(\Delta\boldsymbol\phi=\boldsymbol\phi(A)-\boldsymbol\phi(B)\)，\(y=1\) 表示选了 A。线性奖励模型下 BT 的负对数似然就是以 \(\Delta\boldsymbol\phi\) 为输入的逻辑回归：

Write \(\Delta\boldsymbol\phi=\boldsymbol\phi(A)-\boldsymbol\phi(B)\) and \(y=1\) when A was chosen. With a linear reward model, BT's negative log-likelihood is logistic regression with input \(\Delta\boldsymbol\phi\):

Hessian \(\sum_i\sigma(1-\sigma)\Delta\boldsymbol\phi_i\Delta\boldsymbol\phi_i^\top+2\lambda I\) 正定，FIG 5 每次用牛顿法几步就收敛。BT 等价于假设人看到的分数带有独立的 Gumbel 噪声后取较大者（随机效用模型），它推出“偏好可传递”：若 \(P(A\succ B)>\tfrac12\) 且 \(P(B\succ C)>\tfrac12\) 则 \(P(A\succ C)>\tfrac12\)。真实的人、尤其是多个标注员的集合，常常违反这一点，这是第 9 节一个误区的来源。对神经网络奖励模型（RLHF 里通常是一个去掉输出头、加一个标量头的语言模型），损失的形式完全相同，只是 \(\psi^\top\boldsymbol\phi\) 换成了网络输出。

The Hessian \(\sum_i\sigma(1-\sigma)\Delta\boldsymbol\phi_i\Delta\boldsymbol\phi_i^\top+2\lambda I\) is positive definite, and FIG 5's Newton iteration converges in a few steps each time. BT is equivalent to assuming that the person sees each score plus independent Gumbel noise and picks the larger (a random-utility model), and it implies that preferences are transitive: if \(P(A\succ B)>\tfrac12\) and \(P(B\succ C)>\tfrac12\) then \(P(A\succ C)>\tfrac12\). Real people, and especially pools of several labellers, often violate this, which is the source of one of the pitfalls in Section 9. For a neural reward model (in RLHF, usually a language model with its output head replaced by a scalar head) the loss has exactly the same form, with \(\psi^\top\boldsymbol\phi\) replaced by the network's output.

对固定的 \(\hat r\)，把目标改写成一个 KL：

For a fixed \(\hat r\), rewrite the objective as a KL divergence:

KL 非负、只在两分布相同时为 0，所以最优解唯一，就是 \(\pi^*\)（这是 Gibbs 变分原理，第 3 节的最大熵模型是 \(\pi_{\mathrm{ref}}\) 均匀时的特例）。取对数：\(\hat r(a)=\beta\log\frac{\pi^*(a)}{\pi_{\mathrm{ref}}(a)}+\beta\log Z\)。把它代入 BT 的似然，\(\beta\log Z\) 在 \(\hat r(a_w)-\hat r(a_l)\) 里抵消，就得到第 6 节的 \(\mathcal L_{\mathrm{DPO}}\)：对奖励模型做 BT 最大似然，和对策略做 DPO，是同一个优化问题的两种参数化。DPO 对 \(\theta\) 的梯度是

KL is non-negative and zero only when the two distributions agree, so the optimum is unique and equals \(\pi^*\) (the Gibbs variational principle; the maximum-entropy model of Section 3 is the special case of a uniform \(\pi_{\mathrm{ref}}\)). Taking logs: \(\hat r(a)=\beta\log\frac{\pi^*(a)}{\pi_{\mathrm{ref}}(a)}+\beta\log Z\). Substitute this into the BT likelihood; \(\beta\log Z\) cancels in \(\hat r(a_w)-\hat r(a_l)\), and out comes the \(\mathcal L_{\mathrm{DPO}}\) of Section 6: BT maximum likelihood for a reward model and DPO for a policy are two parameterizations of the same optimization problem. DPO's gradient with respect to \(\theta\) is

每一对的权重是“隐含奖励把这一对排错的程度”：排错得越厉害，越用力地抬高赢家、压低输家。如果某个动作在数据里从没输过，BT 的最大似然要求它的奖励趋于无穷，DPO 就会把别的动作的概率一路压向 0。这是 Azar 等（2024）提出 IPO 的动机；FIG 6 里一个很小的 L2 正则把它挡住了，但 40 个比较时抓取 6 的“四战全胜”仍然把它的奖励估计推到了最高。

Each pair is weighted by how badly the implied reward ranks it: the worse the error, the harder it lifts the winner and pushes down the loser. If an action never loses in the data, BT maximum likelihood wants its reward to go to infinity, and DPO pushes the other actions' probabilities all the way to 0. This motivated IPO (Azar et al., 2024); in FIG 6 a tiny L2 penalty stops the blow-up, but with 40 comparisons grasp 6's "four wins out of four" still pushes its reward estimate to the top.