Actor-Critic：PPO 与 SACActor-Critic: PPO & SAC

20.3 节的 REINFORCE 已经能训练一个连续动作的高斯策略：采一整段轨迹，用回报 \(G_t\) 乘以 \(\nabla_\theta\log\pi_\theta(a_t|s_t)\) 求平均。它是无偏的，但在机器人任务上几乎用不起来，原因有三个：

• 方差太大。一条腿式机器人的轨迹有上千步，\(G_t\) 混进了之后每一步的随机性：脚下一滑、一次碰撞，都会让同一个动作得到完全不同的分数。要把噪声平均掉，需要海量轨迹。
• 步长难定。梯度只在当前策略附近有意义。步子小了学得慢，步子大了策略一下子变坏，而变坏的策略又采回更坏的数据，可能再也爬不回来。
• 数据只用一次。策略梯度要求数据来自当前策略，策略一更新，旧数据就作废了。在仿真里还能忍；在真机上，每一个样本都是电机磨损和实验员的时间。

Actor-Critic 的思路是再训练一个评论家：一个估计价值 \(V_\phi(s)\) 或 \(Q_\phi(s,a)\) 的网络（20.1–20.2 节的价值函数，换成神经网络并在线学习）。演员（actor，也就是策略 \(\pi_\theta\)）用评论家给出的"这一步比平均好多少"（优势）来更新，而不是用整段回报。在这个框架上，两个算法分别解决了后两个问题：

• PPO（Proximal Policy Optimization，Schulman 等，2017）：同策略（on-policy），用"截断的概率比"把每次更新限制在旧策略附近，可以放心地在同一批数据上做多轮梯度；数据多就用并行环境去采。这是 Isaac Gym / Isaac Lab 里训练腿式机器人的主力。
• SAC（Soft Actor-Critic，Haarnoja 等，2018）：异策略（off-policy），把所有经验存进重放缓冲区反复使用；目标里加上策略的熵，让探索自动发生；用两个评论家取较小值，压住 Q 值的高估。它的样本效率高，适合直接在真机上学习。

本节用同一个小任务贯穿全篇：一个受重力作用的单关节手臂，必须在 2 秒内转到随机给定的目标角度并保持住（这需要学会重力补偿），动作是归一化的电机力矩 \(a\in[-1,1]\)。PPO 和 SAC 都在浏览器里从零开始训练它。

Lesson 20.3's REINFORCE can already train a Gaussian policy for continuous actions: sample whole trajectories and average the return \(G_t\) times \(\nabla_\theta\log\pi_\theta(a_t|s_t)\). It is unbiased, but on robot tasks it is nearly unusable, for three reasons:

• The variance is huge. A legged robot's trajectory has thousands of steps, and \(G_t\) mixes in the randomness of every later step: a slipping foot or a collision gives the same action a completely different score. Averaging that noise away takes an enormous number of trajectories.
• The step size is hard to choose. The gradient is only meaningful near the current policy. Small steps learn slowly; a large step can ruin the policy, the ruined policy collects even worse data, and it may never climb back.
• Data is used once. The policy gradient needs data from the current policy, so every update throws the old data away. In simulation that is tolerable; on hardware every sample costs motor wear and an experimenter's time.

The actor-critic idea is to train a second network, the critic, that estimates a value \(V_\phi(s)\) or \(Q_\phi(s,a)\) (the value functions of Lessons 20.1–20.2, now neural networks learned online). The actor (the policy \(\pi_\theta\)) is updated with the critic's estimate of "how much better than average this step was" (the advantage) instead of the whole return. On top of this framework, two algorithms address the last two problems:

• PPO (Proximal Policy Optimization, Schulman et al., 2017) is on-policy. A "clipped probability ratio" keeps each update close to the old policy, so it can safely take several gradient epochs on one batch, and it gets large batches from parallel environments. It is the workhorse for training legged robots in Isaac Gym / Isaac Lab.
• SAC (Soft Actor-Critic, Haarnoja et al., 2018) is off-policy. It keeps all experience in a replay buffer and reuses it, adds the policy's entropy to the objective so that exploration happens by itself, and takes the smaller of two critics to hold down Q-value overestimation. It is sample-efficient and suits learning directly on real robots.

One small task runs through the whole page: a single arm joint under gravity must turn to a random target angle within 2 seconds and hold it there (which requires learning gravity compensation); the action is a normalized motor torque \(a\in[-1,1]\). PPO and SAC both train it from scratch in your browser.

20.3 节已经看到，从回报里减去一个只依赖状态的基线 \(b(s_t)\) 不改变梯度的期望，却能大大降低方差，最好的基线接近 \(V^\pi(s_t)\)。Actor-Critic 把这件事推进一步：既然有了一个学出来的 \(V_\phi\)，就连回报本身也可以用它来估计。最短的版本只看一步，叫时序差分误差（TD error，20.2 节）：

Lesson 20.3 showed that subtracting a state-only baseline \(b(s_t)\) from the return leaves the expected gradient unchanged and cuts the variance a lot, and that the best baseline is close to \(V^\pi(s_t)\). Actor-critic takes one more step: once we have a learned \(V_\phi\), we can use it to estimate the return itself. The shortest version looks one step ahead and is the temporal-difference (TD) error of Lesson 20.2:

"这一步拿到的奖励，加上评论家对下一状态的估值，减去评论家对这一状态的估值"：如果 \(V_\phi=V^\pi\)，它的期望正好是优势 \(A^\pi(s_t,a_t)=Q^\pi(s_t,a_t)-V^\pi(s_t)\)。它只含一步的随机性，方差很小；代价是一旦评论家估错，误差会原样漏进来，产生偏差。另一头是蒙特卡洛：把整段回报 \(G_t\) 减去 \(V_\phi(s_t)\)，没有偏差（评论家只当基线用），但方差最大。广义优势估计 GAE（Schulman 等，2016）用一个旋钮 \(\lambda\in[0,1]\) 在两者之间连续插值：

"The reward of this step, plus the critic's value of the next state, minus its value of this state": if \(V_\phi=V^\pi\), its expectation is exactly the advantage \(A^\pi(s_t,a_t)=Q^\pi(s_t,a_t)-V^\pi(s_t)\). It contains the randomness of one step only, so its variance is small; the price is that any error in the critic leaks straight in as bias. At the other end is Monte Carlo: the whole return \(G_t\) minus \(V_\phi(s_t)\), unbiased (the critic serves only as a baseline) but with the largest variance. Generalized advantage estimation, GAE (Schulman et al., 2016), interpolates continuously between the two with one knob \(\lambda\in[0,1]\):

下图用一个能精确算出真值的小例子来看这个折中。一个送货机器人在 10 格走廊里，从第 2 格出发去第 9 格的充电桩：每走一步奖励 \(-1\)，到达充电桩额外 \(+10\)；它的策略以 0.6 的概率向右、0.4 的概率向左，每次移动有 0.2 的概率打滑原地不动，\(\gamma=0.95\)。我们要估计"在起点向右走"的价值 \(Q^\pi(s_0,\rightarrow)\)，等价于估计它的优势（两者只差一个常数 \(V_\phi(s_0)\)）。评论家 \(V_\phi\) 等于真值加上每格一个固定的随机误差（标准差 \(\sigma_V\)）。估计量的均值和方差都是精确算出来的（两条 Bellman 式递推，见第 5 节），直方图是 1500 条采样轨迹的实际估计值。

The figure looks at this trade-off on a small example whose true values can be computed exactly. A delivery robot in a 10-cell corridor starts in cell 2 and heads for the charging dock in cell 9: every step gives reward \(-1\) and arriving at the dock gives an extra \(+10\). Its policy goes right with probability 0.6 and left with 0.4, every move slips (the robot stays put) with probability 0.2, and \(\gamma=0.95\). We estimate the value of "going right at the start", \(Q^\pi(s_0,\rightarrow)\), which is the same as estimating its advantage (the two differ by the constant \(V_\phi(s_0)\)). The critic \(V_\phi\) is the true value plus a fixed random error in each cell (standard deviation \(\sigma_V\)). The estimator's mean and variance are computed exactly (two Bellman-style recursions, Section 5); the histogram shows the actual estimates from 1500 sampled trajectories.

有了优势估计 \(\hat A_t\)，最直接的做法是对 \(\hat A_t\log\pi_\theta(a_t|s_t)\) 做一步梯度上升，然后把数据扔掉。这太浪费：采数据很贵，更新却很便宜。能不能在同一批数据上多走几步？问题是数据来自旧策略 \(\pi_{\theta_k}\)，策略一变，"新策略下的期望"就要靠重要性采样（4.9 节）来纠正，权重就是概率比（这里记作 \(\rho\)，因为 \(r\) 已经表示奖励；PPO 原文记作 \(r_t\)）：

With advantage estimates \(\hat A_t\) in hand, the most direct move is one gradient-ascent step on \(\hat A_t\log\pi_\theta(a_t|s_t)\), after which the data is discarded. That is wasteful: collecting data is expensive and updates are cheap. Can we take several steps on the same batch? The catch is that the data came from the old policy \(\pi_{\theta_k}\); once the policy changes, expectations under the new policy have to be corrected with importance sampling (Lesson 4.9), whose weight is the probability ratio (written \(\rho\) here because \(r\) is the reward; the PPO paper calls it \(r_t\))

\(L^{\text{PG}}\) 在 \(\theta=\theta_k\) 处的梯度正好是策略梯度（\(\nabla\rho_t=\nabla\log\pi_\theta\)），但离旧策略越远，它越是建立在少数几个权重很大的样本上，估计值就越不可信，而优化器恰恰会利用这种不可信：哪里估计虚高就往哪里冲。PPO 的 \(L^{\text{CLIP}}\) 做了一件很简单的事：一个样本的概率比一旦超出 \([1-\epsilon,1+\epsilon]\)，并且继续变化只会让目标"更好看"，它对梯度的贡献就变成零。好动作的概率最多涨到 \(1+\epsilon\) 倍，坏动作的概率最多降到 \(1-\epsilon\) 倍；反方向（把坏动作变得更可能）则照常受罚，不被截断。

下图是一个只有动作没有状态的例子，正好能把"真实的改进"精确算出来。动作 \(a\) 是推关节的力度：推得越猛越好，可是超过 0.9 会触发安全停机，优势骤降 3。旧策略是 \(\mathcal N(0,0.4^2)\)，它的 16 个样本里没有一个落在停机区，所以数据根本看不见这个悬崖。新策略 \(\mathcal N(m,0.4^2)\)，拖动琥珀色手柄（或在右图里点击）改变 \(m\)。

At \(\theta=\theta_k\) the gradient of \(L^{\text{PG}}\) is exactly the policy gradient (\(\nabla\rho_t=\nabla\log\pi_\theta\)), but the further the policy moves, the more the estimate rests on a few samples with huge weights and the less it can be trusted, and the optimizer exploits exactly that: it rushes to wherever the estimate is inflated. PPO's \(L^{\text{CLIP}}\) does something very simple: once a sample's ratio leaves \([1-\epsilon,1+\epsilon]\) in the direction that would make the objective look even better, that sample stops contributing gradient. A good action's probability can grow by at most \(1+\epsilon\), a bad action's can shrink to at most \(1-\epsilon\) of its old value; the opposite direction (making a bad action more likely) is still penalized in full and never clipped.

The figure is an example with actions but no states, so the true improvement can be computed exactly. The action \(a\) is how hard to push the joint: the harder the better, but beyond 0.9 a safety stop trips and the advantage drops by 3. The old policy is \(\mathcal N(0,0.4^2)\), and none of its 16 samples lands in the stop zone, so the data cannot see the cliff at all. The new policy is \(\mathcal N(m,0.4^2)\): drag the amber handle (or click in the right panel) to change \(m\).

在 \(\epsilon=0.2\) 时 \(L^{\text{CLIP}}\) 的最高点在 \(m\approx0.15\)（KL 约 0.07），真实改进约 +0.25；真正的最优在 \(m\approx0.30\)（+0.36）。不截断的 \(L^{\text{PG}}\) 却在 \(m\approx0.53\) 达到 +0.66，那里的真实改进只剩 +0.09；到 \(m=0.8\) 它仍许诺 +0.52，真实值已是 \(-0.99\)。截断给出的是一个保守但可靠的小步；多走几步，用新数据再截断，比一步跳到估计值的顶点安全得多。

现在把这一切放进完整的算法里。PPO 的一次迭代：(1) 用当前策略在 \(N\) 个并行环境里各跑 \(H=40\) 步；(2) 用评论家算 GAE 优势，在这一批里归一化；(3) 把这 \(NH\) 个样本打乱成大小 128 的小批量，在 \(L^{\text{CLIP}}\) 和评论家的回归损失上做 \(K\) 轮 Adam；(4) 扔掉数据，回到 (1)。演员和评论家都是 3→32→32→1 的 tanh 网络，策略是均值由网络给出、标准差是一个可学习参数的高斯分布，\(\gamma=0.97\)，\(\lambda=0.95\)。每次迭代后，用确定性策略（取均值）在 24 组固定的（起点，目标）上评估：曲线是 2 秒结束时与目标的平均角度误差。

At \(\epsilon=0.2\) the top of \(L^{\text{CLIP}}\) is at \(m\approx0.15\) (KL about 0.07), where the true improvement is about +0.25; the true optimum is at \(m\approx0.30\) (+0.36). The unclipped \(L^{\text{PG}}\) instead peaks at \(m\approx0.53\) with +0.66, where the true improvement is only +0.09; at \(m=0.8\) it still promises +0.52 while the truth is \(-0.99\). Clipping gives a conservative but reliable small step; several such steps, each with fresh data and its own clipping, are far safer than one jump to the top of the estimate.

Now put this into the full algorithm. One PPO iteration: (1) run the current policy for \(H=40\) steps in each of \(N\) parallel environments; (2) compute GAE advantages with the critic and normalize them over the batch; (3) shuffle the \(NH\) samples into minibatches of 128 and run \(K\) epochs of Adam on \(L^{\text{CLIP}}\) and on the critic's regression loss; (4) discard the data and go back to (1). Actor and critic are 3→32→32→1 tanh networks; the policy is a Gaussian whose mean comes from the network and whose standard deviation is a learned parameter; \(\gamma=0.97\), \(\lambda=0.95\). After each iteration the deterministic policy (its mean) is evaluated on 24 fixed (start, target) pairs: the curve is the mean angle error to the target at the end of the 2 seconds.

PPO 的每个样本只服务于采它的那个策略。另一条路是学 \(Q(s,a)\)：Bellman 方程 \(Q(s,a)=r+\gamma\,\mathbb E[V(s')]\) 对任何行为策略收集的转移 \((s,a,r,s')\) 都成立（20.2 节的 Q-learning 正是这样），所以旧数据可以存进重放缓冲区一直用下去。DQN 在离散动作上取 \(\max_a\)；连续动作取不了 max，就让演员网络去找让 \(Q\) 大的动作，这就是 DDPG（Lillicrap 等，2016）和它的改进 TD3（Fujimoto 等，2018）的做法。SAC 在此之上改了目标本身：

Each PPO sample serves only the policy that collected it. The other road is to learn \(Q(s,a)\): the Bellman equation \(Q(s,a)=r+\gamma\,\mathbb E[V(s')]\) holds for transitions \((s,a,r,s')\) collected by any behaviour policy (that is how Q-learning works in Lesson 20.2), so old data can live in a replay buffer and be reused indefinitely. DQN takes \(\max_a\) over discrete actions; with continuous actions there is no max to take, so an actor network learns to find actions with large \(Q\). That is DDPG (Lillicrap et al., 2016) and its refinement TD3 (Fujimoto et al., 2018). SAC goes further and changes the objective itself:

每一步除了奖励，还奖励策略的熵 \(\mathcal H\)（4.11 节）：在同样好的动作之间保持随机。温度 \(\alpha\) 决定熵值多少钱。这个"软"目标的最优策略是 Q 值的玻尔兹曼分布：\(\alpha\to0\) 时退化为 \(\arg\max\)，\(\alpha\) 很大时接近均匀。SAC 的演员是一个 tanh 压缩的高斯（输出 \(\mu,\log\sigma\)，\(a=\tanh(\mu+\sigma\varepsilon)\)），它在每个状态上最小化 \(\mathrm{KL}\big(\pi\,\|\,\pi^\star\big)\)，等价于最小化 \(\mathbb E[\alpha\log\pi(a|s)-Q(s,a)]\)，梯度靠重参数化（19.1 节）穿过采样。

下图把状态拿掉，只看一个固定的 \(Q(a)\)：抓取一个物体有两个好的接近角度，左边的稍好（1.0），右边的稍差（0.8）。拖动 \(\alpha\)，看玻尔兹曼目标怎么变；按"训练"，看单峰的演员怎样去拟合它。

Besides the reward, every step also rewards the policy's entropy \(\mathcal H\) (Lesson 4.11): stay random among equally good actions. The temperature \(\alpha\) sets the price of entropy. The optimal policy of this "soft" objective is the Boltzmann distribution of the Q-values: it tends to \(\arg\max\) as \(\alpha\to0\) and becomes nearly uniform for large \(\alpha\). SAC's actor is a tanh-squashed Gaussian (it outputs \(\mu,\log\sigma\), and \(a=\tanh(\mu+\sigma\varepsilon)\)); in each state it minimizes \(\mathrm{KL}\big(\pi\,\|\,\pi^\star\big)\), which is the same as minimizing \(\mathbb E[\alpha\log\pi(a|s)-Q(s,a)]\), with the gradient passed through the sampling by reparameterization (Lesson 19.1).

The figure drops the state and looks at one fixed \(Q(a)\): there are two good approach angles for grasping an object, the left one slightly better (1.0) and the right one slightly worse (0.8). Drag \(\alpha\) to see how the Boltzmann target changes; press "Train" to watch a single-peaked actor fit it.

两件事值得记住。第一，\(\mathrm{KL}(\pi\|\pi^\star)\) 是寻模（mode-seeking）的：单峰的演员只会选一个峰，从右边出发就停在右边（\(\alpha=0.05\) 时 KL ≈ 3.7，而停在左峰时低于 0.1），这是 tanh 高斯表达能力的限制，也是 19.3 节提到用 flow 做策略的动机。第二，好的 \(\alpha\) 取决于奖励的尺度：奖励乘以 10，同样的 \(\alpha\) 就相当于小了 10 倍。这正是自动温度的用处：与其猜 \(\alpha\)，不如指定"想要多少熵"（默认 \(\bar{\mathcal H}=-\dim\mathcal A\)），让 \(\alpha\) 自己去满足它。

SAC 的第二个部件对付 20.2 节的老问题：高估。评论家有误差，演员又专门去找 Q 最大的动作，于是它找到的往往是误差最大的地方，\(\mathbb E[\max_a\hat Q]\ge\max_a\mathbb E[\hat Q]\)。这个虚高被写进下一次的自举目标，一步步累积。TD3 和 SAC 的对策是训练两个独立初始化的评论家，目标里取较小值。下图把这件事单独拿出来看：真实的 \(Q(s',a)\) 是一条最大值为 1 的抛物线，两个评论家各自加上一条随机的平滑误差曲线（标准差 \(\sigma\)，"粗糙度"是误差曲线的最高频率）。

Two things are worth remembering. First, \(\mathrm{KL}(\pi\|\pi^\star)\) is mode-seeking: a single-peaked actor picks one peak, and starting on the right it stays on the right (at \(\alpha=0.05\) the KL is about 3.7, against less than 0.1 on the left peak). That is a limit of the tanh-Gaussian's expressiveness, and the motivation for flow-based policies mentioned in Lesson 19.3. Second, a good \(\alpha\) depends on the scale of the reward: multiply the reward by 10 and the same \(\alpha\) is effectively 10 times smaller. That is what automatic temperature is for: instead of guessing \(\alpha\), specify how much entropy you want (by default \(\bar{\mathcal H}=-\dim\mathcal A\)) and let \(\alpha\) adjust to deliver it.

SAC's second component addresses an old problem from Lesson 20.2: overestimation. The critic has errors and the actor deliberately seeks the action with the largest Q, so it tends to find where the error is largest: \(\mathbb E[\max_a\hat Q]\ge\max_a\mathbb E[\hat Q]\). The inflated value is written into the next bootstrap target and accumulates step by step. TD3 and SAC train two independently initialized critics and take the smaller one in the target. The figure isolates this effect: the true \(Q(s',a)\) is a parabola with maximum 1, and each of two critics adds its own random smooth error curve (standard deviation \(\sigma\); "roughness" is the highest frequency in the error curve).

高估来自"挑选"：误差越崎岖，演员越容易挑中一个虚高的尖峰（\(\sigma=0.3\)、粗糙度 12 时单评论家平均高估约 0.30，双评论家约 \(-0.12\)）。取两个独立估计的较小值把它压了下去，代价是系统性地偏低：两个独立 \(\mathcal N(0,\sigma^2)\) 误差的较小值，期望是 \(-\sigma/\sqrt\pi\approx-0.56\sigma\)。低估比高估好，因为演员不会去"利用"一个被低估的动作。真实系统里两个评论家用同样的数据和目标训练，误差是相关的，低估比这里温和。

最后把完整的 SAC 放到同一个关节任务上。只有一个环境（像一台真机），前 400 步随机动作预热，之后每走一步就从重放缓冲区（容量 20 000）抽 32 个转移，做一次评论家更新和一次演员更新；目标网络用 \(\tau=0.02\) 的 Polyak 平均跟随；自动温度，目标熵 \(-1\)；网络大小与 PPO 相同。右图同时画出 PPO 在 \(N=16\) 与 \(N=64\) 下的参考曲线（种子 1，与 FIG 3 同一份代码），以及你在 FIG 3 里最近一次的 PPO 训练。

Overestimation comes from selection: the more rugged the error, the more easily the actor picks an inflated spike (at \(\sigma=0.3\) and roughness 12 the single critic overestimates by about 0.30 on average, the twin by about \(-0.12\)). Taking the smaller of two independent estimates pushes this down, at the price of a systematic low bias: the minimum of two independent \(\mathcal N(0,\sigma^2)\) errors has expectation \(-\sigma/\sqrt\pi\approx-0.56\sigma\). Underestimation is the lesser evil because the actor never "exploits" an underestimated action. In a real system the two critics are trained on the same data and targets, so their errors are correlated and the underestimation is milder than here.

Finally, run full SAC on the same joint task. There is one environment (like a real robot). The first 400 steps use random actions as warm-up; after that, every environment step draws 32 transitions from the replay buffer (capacity 20,000) for one critic update and one actor update; target networks follow by Polyak averaging with \(\tau=0.02\); the temperature is automatic with target entropy \(-1\); the networks are the same size as PPO's. The right panel also shows PPO reference curves for \(N=16\) and \(N=64\) (seed 1, the same code as FIG 3) and your latest PPO run from FIG 3.

FIG 6 的结论可以推广：先问数据从哪里来、付的是样本还是时间。

The conclusion of FIG 6 generalizes: first ask where the data comes from, and whether you pay in samples or in time.