Part V · 第 20 章强化学习 · 20.3Part V · Ch. 20 Reinforcement Learning · 20.3

策略梯度 (Policy Gradients)Policy Gradients

不学价值表，直接调控制器：把策略写成带参数的概率分布 $\pi_\theta(a\mid s)$，让它多做带来高回报的动作、少做带来低回报的动作。对数导数技巧让我们只靠试出来的回报就能估计梯度，不需要环境模型，也不需要奖励可导。本节在浏览器里实时训练：一个高斯策略学会抓取偏移，REINFORCE 在不知道动力学的情况下找回一个关节的 LQR 增益，再看这个估计为什么噪声很大、reward-to-go、基线和优势函数怎样把方差压下来，它和参数空间里的进化策略（6.11 节）是什么关系，最后用一个神经网络策略控制受重力的机械臂。 Instead of learning a table of values, tune the controller directly: write the policy as a parametrized probability distribution $\pi_\theta(a\mid s)$ and make the actions that led to high return more likely and those that led to low return less likely. The log-derivative trick estimates the gradient from nothing but the returns of trial runs, with no model of the environment and no derivative of the reward. In this lesson everything trains live in your browser: a Gaussian policy learns a grasp offset, REINFORCE recovers the LQR gain of a joint without knowing its dynamics, and then we see why the estimate is so noisy, how reward-to-go, baselines and the advantage bring the variance down, how it relates to evolution strategies in parameter space (Lesson 6.11), and finally a neural-network policy that controls an arm under gravity.

约 45 分钟~45 min 前置：20.1 MDP 与 Bellman 方程 · 4.9 采样方法 · 6.11 无梯度与采样优化 · 16.3 反向传播 · 对照 20.2 Q-learningPrereqs: 20.1 MDPs & Bellman equations · 4.9 Sampling methods · 6.11 Derivative-free & sampling-based optimization · 16.3 Backpropagation · for contrast 20.2 Q-learning 5 个交互5 interactives

1为什么机器人需要它Why robots need this

20.2 节的 Q-learning 和 DQN 先学 $Q(s,a)$，再在每一步取 $\arg\max_a Q(s,a)$。动作只有几个离散选项时这很自然；可机器人的动作是连续的：一个 7 关节机械臂每一步要给出 7 个力矩，在 $\R^7$ 里求最大值本身就是一个优化问题，每个控制周期都要解一次。而且机器人工程师手里的控制器本来就是“带参数的策略”：PD 增益（13.1 节）、步态参数、一个神经网络。直接调这些参数、让任务回报变大，叫作策略搜索（policy search）。

难点在于回报对参数的依赖要经过整个环境：仿真器里的接触、真实的电机、一个“抓住了 / 没抓住”的二值结果。这些我们没法求导，常常连模型都没有。策略梯度绕开了这一点：只要策略是随机的，回报的期望就是参数的光滑函数，它的梯度可以只用“试了哪些动作、得了多少回报”来估计，不需要环境的导数，也不需要奖励的导数。这就是 REINFORCE（Williams 1992），后面的 actor-critic、PPO（20.4 节）和今天腿式机器人的大规模仿真训练（23.1 节）都建立在它上面。

本节的顺序：先在一个一步就结束的任务上看清“对数导数技巧”在做什么；再让 REINFORCE 调一个关节的 PD 增益，看它在不知道模型的情况下找到 LQR 的答案；然后正面处理它最大的问题——方差，以及它和 6.11 节进化策略的关系；最后用神经网络策略控制一条受重力的机械臂。

The Q-learning and DQN of Lesson 20.2 first learn $Q(s,a)$ and then take $\arg\max_a Q(s,a)$ at every step. That is natural when there are a few discrete actions; but a robot's actions are continuous: a 7-joint arm outputs 7 torques per step, and maximizing over $\R^7$ is an optimization problem in its own right, to be solved every control cycle. Besides, the controllers a robotics engineer already has are parametrized policies: PD gains (Lesson 13.1), gait parameters, a neural network. Tuning those parameters directly so that the task return grows is called policy search.

The hard part is that the return depends on the parameters through the whole environment: contacts in a simulator, real motors, a binary "grasped / not grasped" outcome. We cannot differentiate through any of that and often have no model at all. Policy gradients sidestep this: as long as the policy is stochastic, the expected return is a smooth function of the parameters, and its gradient can be estimated from nothing but "which actions were tried and what return they got", with no derivative of the environment and none of the reward. That is REINFORCE (Williams 1992), and actor-critic methods, PPO (Lesson 20.4) and today's massively parallel training of legged robots in simulation (Lesson 23.1) are all built on it.

The plan: first see what the "log-derivative trick" does on a task that ends after a single step; then let REINFORCE tune the PD gains of a joint and watch it find the LQR answer without knowing the model; then face its biggest problem, variance, head on, together with its relation to the evolution strategies of Lesson 6.11; and finally control an arm under gravity with a neural-network policy.

2直觉：试一试，再让做得好的动作更常出现Intuition: try things, then make what worked more likely

最简单的情形：只有一步，没有状态。机械手要决定抓取的横向偏移 $a$（单位 cm），奖励 $r(a)$ 告诉它抓得怎样。策略是一个高斯分布 $a\sim\mathcal N(\mu,\sigma^2)$，参数是均值 $\mu$ 和 $\log\sigma$。目标是期望奖励 $J(\mu,\sigma)=\mathbb E[r(a)]$。最难的预设是“二值成功”：只有偏移落在 $[1.5,2.5]$ 里才算抓住（$r=1$），否则 $r=0$。这个 $r$ 在几乎所有地方导数都是零，对它求导毫无用处；但 $J$ 是 $\mu$ 的光滑函数，因为改变 $\mu$ 会改变“落进窗口”的概率。

REINFORCE 每轮做三件事：从当前策略抽 $N$ 个动作，看每个动作的奖励；给每个样本一个权重 $r_i-\hl{base}{b}$（$b$ 是基线，这里取其余样本奖励的平均）；然后把每个样本的 $\hl{score}{\log\pi(a_i)}$ 按权重往上推。权重为正的样本（比平均好）概率变大，为负的变小。“把 $\log\pi(a_i)$ 往上推”具体就是：$\mu$ 朝 $a_i$ 移动，移动量正比于 $(a_i-\mu)/\sigma^2$；而 $\log\sigma$ 在样本离均值远于 $\sigma$ 时变大、近于 $\sigma$ 时变小。

The simplest case: one step, no state. A gripper must choose the lateral grasp offset $a$ (in cm), and the reward $r(a)$ says how good the grasp was. The policy is a Gaussian $a\sim\mathcal N(\mu,\sigma^2)$ with parameters $\mu$ and $\log\sigma$, and the objective is the expected reward $J(\mu,\sigma)=\mathbb E[r(a)]$. The hardest preset is "binary success": the grasp holds ($r=1$) only if the offset lands in $[1.5,2.5]$, otherwise $r=0$. The derivative of this $r$ is zero almost everywhere, so differentiating it is useless; but $J$ is a smooth function of $\mu$, because moving $\mu$ changes the probability of landing in the window.

Every round REINFORCE does three things: draw $N$ actions from the current policy and look at each one's reward; give each sample a weight $r_i-\hl{base}{b}$ ($b$ is a baseline, here the average reward of the other samples); then push each sample's $\hl{score}{\log\pi(a_i)}$ up in proportion to its weight. Samples with positive weight (better than average) become more likely, those with negative weight less likely. Concretely, "pushing $\log\pi(a_i)$ up" moves $\mu$ towards $a_i$ by an amount proportional to $(a_i-\mu)/\sigma^2$, and grows $\log\sigma$ when the sample lies farther than $\sigma$ from the mean, shrinks it when closer.

FIG 1一步任务上的 REINFORCE：按权重推高每个样本的概率REINFORCE on a one-step task: push up each sample's probability by its weight

上半部分：蓝线是奖励 $r(a)$，琥珀色是策略的密度（画成峰高为 1），下方的琥珀色括号是 $\mu\pm\sigma$；绿色虚线是基线对应的奖励水平（奖励高于它的样本权重为正）。下半部分：本批 $N$ 个样本，每个样本一根竖线，长度是它的权重 $r_i+c-b$，绿为正、红为负（超过 ±1 的截断并画一个小横杠）；紫色曲线是标准化噪声 $(a-\mu)/\sigma$，即得分 $\partial\log\pi/\partial\mu=(a-\mu)/\sigma^2$ 乘以 $\sigma$（形状相同）。顶部两支箭头按同一个固定比例画出（超过 4 cm 截断）：琥珀是这一批的估计 $\hat g_\mu$，紫是精确的 $\partial J/\partial\mu$；更新时 $\mu$ 实际移动 $\alpha\hat g_\mu$。图上显示的就是下一次更新要用的那一批样本。可以拖动琥珀色手柄改 $\mu$。右上：$J$ 随更新次数的变化（精确值，数值积分得到），绿虚线是这个奖励能达到的最大值 1。 Top half: the blue line is the reward $r(a)$, amber is the policy density (drawn with peak height 1), and the amber bracket below it is $\mu\pm\sigma$; the green dashed line is the reward level of the baseline (samples rewarded above it get positive weight). Bottom half: this batch of $N$ samples, one stem each, its length the sample's weight $r_i+c-b$, green if positive, red if negative (clipped at ±1 with a small bar); the purple curve is the standardized noise $(a-\mu)/\sigma$, i.e. $\sigma$ times the score $\partial\log\pi/\partial\mu=(a-\mu)/\sigma^2$ (same shape). The two arrows at the top are drawn at the same fixed scale (capped at 4 cm): amber is this batch's estimate $\hat g_\mu$, purple the exact $\partial J/\partial\mu$; an update moves $\mu$ by $\alpha\hat g_\mu$. The batch shown is exactly the one the next update will use. Drag the amber handle to move $\mu$. Top right: $J$ against the number of updates (exact, by numerical integration); the green dashed line is the best value this reward allows, 1.

试一试Try this

“二值成功”、默认设置，按“训练”：开始时成功率只有 6%，大约 80 次更新后 $J\ge0.9$。读数里“逐样本求导”的估计始终是 0：奖励的导数帮不上忙。"Binary success" with the defaults, press "Train": the success rate starts at only 6%, and after about 80 updates $J\ge0.9$. The "differentiate each sample" estimate in the readout stays at 0 throughout: the derivative of the reward is no help.
还是“二值成功”，把基线关掉（“无”），重置再训练：它照样找到窗口（$J\ge0.9$），可接下来又被“踢”出去，$J$ 掉回 0.1 以下。原因：每个样本都成功时，没有基线的权重全是 1，$\mu$ 被这些样本的随机位置来回拽。Still "binary success", switch the baseline off ("none"), reset and train: it still finds the window ($J\ge0.9$), but then gets "kicked" out again and $J$ falls back below 0.1. The reason: once every sample succeeds, all weights without a baseline equal 1, and $\mu$ is jerked around by the random positions of those samples.
基线关掉，把奖励偏移 $c$ 拉到 4.5 以上（每个奖励都加上 $c$，最优动作不变）：读数里 $\hat g_\mu$ 的离散程度变成 $c=0$ 时的好几倍（初始策略下 $c=5$ 时约 10 倍）；再把基线打开：离散程度回到 $c=0$ 的水平，与 $c$ 无关。With the baseline off, raise the reward offset $c$ above 4.5 (every reward gets $c$ added; the best action does not change): the spread of $\hat g_\mu$ in the readout becomes several times its value at $c=0$ (about 10 times at $c=5$ for the starting policy); now switch the baseline on: the spread drops back to its $c=0$ level, whatever $c$ is.
选“两个峰”并训练：策略收缩到离起点近的矮峰上（$J\approx0.49$），$\sigma$ 变得很小，梯度也几乎为零。然后把 $\mu$ 拖到 0.5 右边，用滑块把 $\sigma$ 调回 1 左右，再训练：它爬上高峰，$J\ge0.9$。策略梯度是局部方法，探索靠 $\sigma$。Choose "two peaks" and train: the policy shrinks onto the short peak near its start ($J\approx0.49$), $\sigma$ becomes tiny and so does the gradient. Then drag $\mu$ to the right of 0.5, set $\sigma$ back to about 1 with the slider and train again: it climbs the tall peak, $J\ge0.9$. Policy gradient is a local method, and exploration comes from $\sigma$.

核心洞见Key insight

策略梯度求导的对象是概率，不是奖励。它的更新就是一次加权的最大似然（4.6 节）：对抽到的动作做极大似然拟合，只是每个样本的权重是“它比基线好多少”。这也说明了它和行为克隆（22.1 节）的关系：行为克隆是所有权重都等于 1、样本来自专家的同一个更新。

Policy gradient differentiates the probability, not the reward. Its update is a weighted maximum-likelihood step (Lesson 4.6): fit the drawn actions by maximum likelihood, with each sample weighted by how much better than the baseline it was. This also shows its relation to behaviour cloning (Lesson 22.1): behaviour cloning is the same update with every weight equal to 1 and the samples coming from an expert.

3动手：让 REINFORCE 调一个关节的 PD 增益Play: let REINFORCE tune the PD gains of a joint

现在有了状态和时间。一个水平放置的关节（没有重力，单位惯量）要在 3 秒内从离目标 1 rad 的地方转到目标并停住。状态是角度误差和角速度 $s=(e,\dot q)$，动作是力矩 $a$，控制周期 $\Delta t=0.1$ s，共 $T=30$ 步。第 $t$ 步的奖励是负的代价 $r_{t+1}=-\Delta t\,(e_t^2+0.1\,\dot q_t^2+0.01\,a_t^2)$（下标沿用 20.1 节：动作 $a_t$ 换来 $r_{t+1}$），回报是 30 步奖励之和（有限时域，不打折扣）。起点误差每次随机，均值 −1 rad、标准差 0.2 rad。

策略是加了噪声的 PD 控制器：$a=-k_p e-k_d\dot q+\sigma\epsilon$，$\epsilon\sim\mathcal N(0,1)$，参数 $\theta=(k_p,k_d)$。因为系统是线性的、代价是二次的，这个页面能精确算出每组增益的期望回报 $J(\theta)$（传播状态的均值和协方差），画成背景的地形图，还能算出精确梯度 $\hl{grad}{\nabla_\theta J}$ 作对照。REINFORCE 本身完全不用这些：它只看 $N$ 条采样出来的轨迹和它们的奖励，用下面第 5 节的公式算出估计 $\hl{ghat}{\hat g}$，然后做梯度上升 $\theta\leftarrow\theta+\alpha\hat g$。估计方式先选“reward-to-go − 基线”（第 4 节解释），再换成“完整回报”对比。

Now there are states and time. A horizontal joint (no gravity, unit inertia) must turn from 1 rad away to its target and stop there within 3 seconds. The state is the angle error and the joint speed $s=(e,\dot q)$, the action is the torque $a$, the control period is $\Delta t=0.1$ s, $T=30$ steps. The reward of step $t$ is a negative cost $r_{t+1}=-\Delta t\,(e_t^2+0.1\,\dot q_t^2+0.01\,a_t^2)$ (indices as in Lesson 20.1: action $a_t$ earns $r_{t+1}$), and the return is the sum of the 30 rewards (finite horizon, no discount). The starting error is random each time, with mean −1 rad and standard deviation 0.2 rad.

The policy is a PD controller with noise: $a=-k_p e-k_d\dot q+\sigma\epsilon$, $\epsilon\sim\mathcal N(0,1)$, with parameters $\theta=(k_p,k_d)$. Because the system is linear and the cost quadratic, this page can compute the expected return $J(\theta)$ of every pair of gains exactly (by propagating the mean and covariance of the state), draw it as the background landscape, and compute the exact gradient $\hl{grad}{\nabla_\theta J}$ for comparison. REINFORCE itself uses none of that: it only sees $N$ sampled trajectories and their rewards, computes the estimate $\hl{ghat}{\hat g}$ with the formula of Section 5, and climbs $\theta\leftarrow\theta+\alpha\hat g$. Start with the estimator "reward-to-go − baseline" (explained in Section 4), then switch to "full return" to compare.

FIG 2REINFORCE 在增益空间里爬坡：不知道模型，也能找到 LQR 增益REINFORCE climbing in gain space: it finds the LQR gain without a model

左：横轴 $k_p$，纵轴 $k_d$，背景是精确的期望回报 $J(\theta)$，越青 = 回报越高（分成 10 档，最低一档包括所有低于 −3.5 的区域）。绿色星号是用模型解出的 LQR 增益（13.3 节），它就是这张图上的最高点（$\sigma=0.5$ 时两者相差不到 0.03）。琥珀色折线是 $\theta$ 走过的路，琥珀点是当前位置（暂停时可拖动它换起点）；从它出发的琥珀箭头是这一步实际走的 $\alpha\hat g$，紫箭头是精确梯度会走的 $\alpha\nabla_\theta J$。右上：最近一批 $N$ 条轨迹的角度误差 $e(t)$（琥珀细线），目标是 0。右下：$J(\theta)$ 随更新次数（琥珀线），每批采样回报的平均（青点），绿虚线是 LQR 增益的 $J$。改变 $\sigma$ 会改变整张地形（噪声本身也有代价），但最高点几乎不动（$\sigma$ 从 0.2 到 1.5，$k_p$ 只从 7.60 移到 7.52）。 Left: $k_p$ across, $k_d$ up, and the background is the exact expected return $J(\theta)$, more cyan = higher return (10 bands; the lowest band holds everything below −3.5). The green star is the LQR gain solved from the model (Lesson 13.3), which is the top of this map (within 0.03 at $\sigma=0.5$). The amber polyline is the path of $\theta$ and the amber dot its current position (drag it while paused to choose a new start); the amber arrow from it is the step $\alpha\hat g$ actually taken, the purple arrow the step $\alpha\nabla_\theta J$ the exact gradient would take. Top right: the angle error $e(t)$ of the latest batch of $N$ trajectories (thin amber lines); the target is 0. Bottom right: $J(\theta)$ against the update count (amber line), the average sampled return of each batch (cyan dots), and the green dashed line is $J$ at the LQR gain. Changing $\sigma$ changes the whole landscape (the noise has a cost of its own), but its top barely moves (from $\sigma=0.2$ to 1.5, $k_p$ only shifts from 7.60 to 7.52).

默认设置（reward-to-go − 基线，$N=10$，$\alpha=2$）按“训练”：不到 20 次更新（本页的随机种子下是 17 次）$J$ 就进入最优值 0.05 以内，路径停在绿色星号附近，而算法从没见过动力学方程。With the defaults (reward-to-go − baseline, $N=10$, $\alpha=2$) press "Train": in fewer than 20 updates (17 with this page's random seed) $J$ gets within 0.05 of the optimum and the path settles near the green star, although the algorithm never saw the equations of motion.
把估计方式换成“完整回报 $G_0$”，其余不变，再训练至少 100 次更新：路径在地形上乱走，最近 50 次更新的平均 $J$ 比最优值差 0.3 以上。同样的期望方向，噪声却大得多。Switch the estimator to "full return $G_0$", change nothing else, and train for at least 100 more updates: the path wanders across the landscape, and the average $J$ over the last 50 updates stays more than 0.3 below the optimum. Same expected direction, far more noise.

有意思的细节Things worth noticing

(1) 星号处的增益 $(7.60,\,4.22)$ 是 13.3 节的离散 LQR 用模型和 Riccati 方程解出来的；REINFORCE 只用试验就走到了同一个地方。代价是样本：每次更新 10 条轨迹，走到星号附近就用了一两百条，之后在山脊上的每一步还要继续花，而 LQR 一条都不需要。 (2) 起点 $(1,0.2)$ 处 $J=-1.37$，最优值 $J^\star=-0.589$（$\sigma=0.5$）；最优附近是一条又长又平的山脊，精确梯度很小，噪声的影响就相对变大，所以路径到了山脊上会来回晃。 (3) 学习率 $\alpha$ 在这里是“回报单位”的：奖励放大 10 倍，同样的 $\alpha$ 步长也放大 10 倍。这是实际实现里普遍把优势归一化（FIG 5）或用 Adam 的原因之一。

(1) The gain at the star, $(7.60,\,4.22)$, is what the discrete LQR of Lesson 13.3 computes from the model with the Riccati equation; REINFORCE walks to the same place using trials alone. The price is samples: 10 trajectories per update, one to two hundred to get near the star and more for every step along the ridge afterwards, while LQR needs none. (2) At the start $(1,0.2)$ the return is $J=-1.37$, the optimum is $J^\star=-0.589$ (for $\sigma=0.5$); around the optimum lies a long, flat ridge where the exact gradient is small, so the noise matters relatively more and the path jiggles once it reaches the ridge. (3) The learning rate $\alpha$ is "in units of return" here: multiply the rewards by 10 and the same $\alpha$ takes a 10 times larger step. That is one reason real implementations normalize the advantages (FIG 5) or use Adam.

4另外两个视角：方差从哪来，以及在哪个空间里探索Two more views: where the variance comes from, and where to explore

方差的阶梯：reward-to-go、基线、价值函数The variance ladder: reward-to-go, baselines, the value function

FIG 2 里“完整回报”和“reward-to-go − 基线”在期望上给出同一个梯度，差别全在噪声。下图把它量出来：固定一组增益，用同一个 $N$ 重复估计 200 次，把每次的 $\hat g$ 画成梯度平面上的一个点，再和精确梯度比较。四种估计一个比一个好，每一级都只是去掉了一部分“与这个动作无关”的随机性：

完整回报 $G_0$：第 $t$ 步动作的得分乘以整条轨迹的回报。可 $t$ 之前的奖励不可能受第 $t$ 步动作影响，它们只贡献噪声。
reward-to-go $G_t$：只乘从 $t$ 开始的奖励 $\hl{rtg}{G_t=\sum_{t'\ge t}r_{t'+1}}$（“因果性”，期望不变，见第 5 节）。
$G_t-b_t$：再减去一个与动作无关的基线，这里是同一批里其余轨迹在第 $t$ 步的 $G_t$ 的平均（留一法，保证无偏）。它去掉的是“这一时刻大家都差不多会拿到的回报”。
$G_t-V(s_t)$：基线换成当前策略的真实价值函数 $V^\pi(s_t)$（20.1 节）。这里线性二次的系统能精确算出它，相当于一个“完美的 critic”。$G_t-V(s_t)$ 是优势 $A(s_t,a_t)=Q(s_t,a_t)-V(s_t)$ 的一个无偏样本：这个动作比“这个状态下的平均水平”好多少。20.4 节的 actor-critic 就是用一个学出来的 $V$ 去近似这一级。

In FIG 2, "full return" and "reward-to-go − baseline" give the same gradient in expectation; the difference is all noise. The figure below measures it: fix a pair of gains, repeat the estimate 200 times with the same $N$, plot each $\hat g$ as a point in the gradient plane and compare with the exact gradient. The four estimators get better one after another, and each step removes only a part of the randomness that "has nothing to do with this action":

Full return $G_0$: the score of the action at step $t$ times the return of the whole trajectory. But rewards before step $t$ cannot be affected by the action at $t$; they only add noise.
Reward-to-go $G_t$: multiply only by the rewards from $t$ on, $\hl{rtg}{G_t=\sum_{t'\ge t}r_{t'+1}}$ ("causality"; the expectation is unchanged, see Section 5).
$G_t-b_t$: also subtract a baseline that does not depend on the action, here the average $G_t$ at step $t$ of the other trajectories in the batch (leave-one-out, which keeps it unbiased). It removes "the return everybody gets at this moment anyway".
$G_t-V(s_t)$: use the true value function $V^\pi(s_t)$ of the current policy as the baseline (Lesson 20.1). This linear-quadratic system lets us compute it exactly, a "perfect critic". $G_t-V(s_t)$ is an unbiased sample of the advantage $A(s_t,a_t)=Q(s_t,a_t)-V(s_t)$: how much better this action was than "the average in this state". The actor-critic methods of Lesson 20.4 approximate this rung with a learned $V$.

FIG 3同一个梯度，四种估计：方差的阶梯One gradient, four estimators: the variance ladder

左：梯度平面（横轴 $\partial J/\partial k_p$，纵轴 $\partial J/\partial k_d$），原点在中心，视野半宽是精确梯度长度的 4 倍。紫箭头是精确梯度，琥珀点是所选估计方式的 200 次独立估计（每次 $N$ 条轨迹），绿十字是它们的平均：四种方式的平均都落在箭头尖上（无偏），只是散开的程度不同。跑出视野的点数写在左上角。右：四种方式在同样设置下的两个指标。上：相对均方误差 $\mathbb E\|\hat g-\nabla J\|^2/\|\nabla J\|^2$（对数刻度；小于 1 意味着误差比梯度本身小）。下：$\hat g$ 与精确梯度夹角余弦的平均（1 = 方向完全正确，0 = 和随机方向一样）。三组增益：A $(1,0.2)$，B $(2,1)$，C $(5,1)$。 Left: the gradient plane ($\partial J/\partial k_p$ across, $\partial J/\partial k_d$ up), origin at the centre, half-width 4 times the length of the exact gradient. The purple arrow is the exact gradient, the amber dots are 200 independent estimates of the selected kind (each from $N$ trajectories), and the green cross is their mean: for all four kinds the mean lands on the arrow tip (unbiased); only the spread differs. The number of dots outside the view is printed top left. Right: two measures for all four kinds at the same settings. Top: the relative mean-squared error $\mathbb E\|\hat g-\nabla J\|^2/\|\nabla J\|^2$ (log scale; below 1 means the error is smaller than the gradient itself). Bottom: the mean cosine between $\hat g$ and the exact gradient (1 = exactly the right direction, 0 = no better than a random direction). Three pairs of gains: A $(1,0.2)$, B $(2,1)$, C $(5,1)$.

在 B 点、$N=10$ 下依次看四种估计：相对误差大约是 88、20、3、0.3，余弦从 0.05 升到 0.94。选“$G_t-V(s_t)$”并保持 $N\le10$，看到余弦 $\ge0.9$。At B with $N=10$, go through the four estimators: the relative error is roughly 88, 20, 3 and 0.3, and the cosine rises from 0.05 to 0.94. Select "$G_t-V(s_t)$" with $N\le10$ and see a cosine of at least 0.9.
在 B 点选“完整回报 $G_0$”，把 $N$ 拉到 100：轨迹多了 10 倍，余弦也只有约 0.31，仍不如 $N=10$ 的“$G_t-b_t$”（约 0.61）。方差按 $1/N$ 下降，减少方差的技巧却能一下子省掉一个数量级的样本。At B choose "full return $G_0$" and push $N$ to 100: ten times the trajectories, yet the cosine is only about 0.31, still worse than "$G_t-b_t$" at $N=10$ (about 0.61). Variance falls like $1/N$, whereas a variance-reduction trick saves an order of magnitude of samples at once.

还有一个旋钮：探索噪声 $\sigma$。得分 $(a-\mu)/\sigma^2$ 的大小正比于 $1/\sigma$，$\sigma$ 越小，每个样本的梯度信号越强，可随机性也越大：B 点、$N=10$ 时，$\sigma$ 从 0.5 降到 0.2，“$G_t-b_t$”的相对误差从 3.0 涨到 16.5；“$G_t-V(s_t)$”却几乎不变（0.28 → 0.26）。好的基线把与动作无关的部分全去掉之后，剩下的才是真正由探索噪声带来的信号。

There is one more knob: the exploration noise $\sigma$. The score $(a-\mu)/\sigma^2$ scales like $1/\sigma$; the smaller $\sigma$, the stronger each sample's gradient signal, but also the larger the randomness: at B with $N=10$, lowering $\sigma$ from 0.5 to 0.2 raises the relative error of "$G_t-b_t$" from 3.0 to 16.5, whereas "$G_t-V(s_t)$" hardly changes (0.28 → 0.26). Once a good baseline has removed everything that does not depend on the action, what remains is the signal that the exploration noise actually carries.

在动作空间还是参数空间里探索：与进化策略的关系Explore in action space or in parameter space: the relation to evolution strategies

6.11 节里有一个恒等式：高斯平滑后的函数的梯度是 $\nabla f_\sigma(\theta)=\tfrac1\sigma\mathbb E[f(\theta+\sigma\boldsymbol\epsilon)\boldsymbol\epsilon]$。把 $f$ 换成“确定性策略 $\theta$ 跑一整条轨迹的回报”，这就是进化策略（ES）：在参数空间里加噪声，每条轨迹只用一组扰动后的增益，整条轨迹不再加任何动作噪声。它其实也是对数导数技巧，只不过“策略”是参数的高斯分布 $\theta'\sim\mathcal N(\theta,\sigma_\theta^2I)$，每条轨迹只做一次“动作”（选一组参数）。CEM 和 MPPI 是同一家族：用回报给参数样本加权平均。

两者的方差来源不同。REINFORCE 每一步都注入新的动作噪声，第 $t$ 步噪声的影响要从后面所有步的奖励里“分辨”出来；控制频率越高，一条轨迹里的噪声越多，每一份的影响越小，这个信用分配问题越难。ES 根本不关心一条轨迹有多少步，只关心参数有几维：方差随参数维数增长。下图在 B 点比较三者，每次估计都用 10 条轨迹（ES 用 5 对正负对称的扰动，每对共用同一个起点）。

Lesson 6.11 has an identity: the gradient of the Gaussian-smoothed function is $\nabla f_\sigma(\theta)=\tfrac1\sigma\mathbb E[f(\theta+\sigma\boldsymbol\epsilon)\boldsymbol\epsilon]$. Take $f$ to be "the return of one whole trajectory of the deterministic policy $\theta$" and you have evolution strategies (ES): noise is added in parameter space, each trajectory uses one perturbed set of gains, and no action noise is added along the way. It is the log-derivative trick too, only the "policy" is a Gaussian over parameters, $\theta'\sim\mathcal N(\theta,\sigma_\theta^2I)$, with a single "action" (pick a parameter vector) per trajectory. CEM and MPPI belong to the same family: average the parameter samples weighted by their return.

Their variance comes from different places. REINFORCE injects fresh action noise at every step, and the effect of the noise at step $t$ has to be "told apart" in the rewards of all later steps; the higher the control rate, the more noise per trajectory, the smaller the effect of each piece, and the harder this credit assignment. ES does not care how many steps a trajectory has, only how many parameters there are: its variance grows with the parameter dimension. The figure compares the three at B, every estimate using 10 trajectories (ES uses 5 antithetic pairs of perturbations, each pair sharing one starting state).

FIG 4动作噪声与参数噪声：控制频率越高，REINFORCE 越难分辨功劳Action noise versus parameter noise: the higher the control rate, the harder REINFORCE's credit assignment

左：三个梯度平面（画法同 FIG 3，半宽是精确梯度的 4 倍），各 200 次估计：REINFORCE 用“$G_t-b_t$”（动作噪声 $\sigma=0.5$），REINFORCE 用精确的 $V(s_t)$ 作基线，以及 ES（参数噪声 $\sigma_\theta=0.5$，扰动后的增益在 0 处截断）。紫箭头是各自目标的精确梯度：REINFORCE 优化带动作噪声的 $J$；ES 优化的是确定性策略的 $J$ 在参数噪声下的高斯平滑 $J_{\sigma_\theta}$（第 5 节，扰动同样在 0 处截断，用数值积分算出）。右：平均余弦随控制频率（5、10、20、50 Hz，即 $\Delta t$ = 0.2、0.1、0.05、0.02 s；总时长都是 3 s）的变化，琥珀是 $G_t-b_t$，绿是 $G_t-V(s_t)$，紫是 ES。频率在后台依次算出。 Left: three gradient planes (drawn as in FIG 3, half-width 4 times the exact gradient), 200 estimates each: REINFORCE with "$G_t-b_t$" (action noise $\sigma=0.5$), REINFORCE with the exact $V(s_t)$ as its baseline, and ES (parameter noise $\sigma_\theta=0.5$, perturbed gains clipped at 0). The purple arrows are the exact gradients of each one's own objective: REINFORCE optimizes $J$ with action noise; ES optimizes the deterministic policy's $J$ Gaussian-smoothed by the parameter noise, $J_{\sigma_\theta}$ of Section 5 (with the same clipping at 0, computed by quadrature). Right: the mean cosine against the control rate (5, 10, 20, 50 Hz, i.e. $\Delta t$ = 0.2, 0.1, 0.05, 0.02 s; always 3 s in total): amber is $G_t-b_t$, green $G_t-V(s_t)$, purple ES. The rates are computed one after another in the background.

把控制频率从 10 Hz 换到 50 Hz：“$G_t-b_t$”的余弦从约 0.61 掉到约 0.35，ES 稳定在约 0.88，带精确 $V$ 的 REINFORCE 保持约 0.94。Switch the control rate from 10 Hz to 50 Hz: the cosine of "$G_t-b_t$" drops from about 0.61 to about 0.35, ES stays at about 0.88, and REINFORCE with the exact $V$ holds at about 0.94.

一个想法，几种看法One idea, several views

蒙特卡洛：$\nabla J$ 是一个期望，用样本平均估计（4.9 节）；整节课的问题都是“怎样让这个平均的方差小”。 得分函数估计：19.1 节比较过的“得分函数（REINFORCE）对重参数化”就是它；策略梯度选择前者，是因为环境不可导。 加权最大似然：对样本做 $\sum_iw_i\log\pi(a_i)$ 的极大似然，权重是优势。平滑：随机策略把一个不可导、甚至不连续的回报变成参数的光滑函数，6.11 节的 ES 在参数空间做同一件事。 行为克隆：所有权重为 1、动作来自专家，就是 22.1 节的 BC。

Monte Carlo: $\nabla J$ is an expectation, estimated by a sample average (Lesson 4.9); the whole lesson is about making that average less variable. Score-function estimator: the "score function (REINFORCE) versus reparameterization" comparison of Lesson 19.1 is exactly this; policy gradients use the former because the environment is not differentiable. Weighted maximum likelihood: maximize $\sum_iw_i\log\pi(a_i)$ over the samples, with the advantages as weights. Smoothing: a stochastic policy turns a non-differentiable, even discontinuous, return into a smooth function of the parameters; the ES of Lesson 6.11 does the same in parameter space. Behaviour cloning: all weights 1 and the actions from an expert is the BC of Lesson 22.1.

5再看数学The math

提示：把鼠标悬停在公式中带底色的符号上，图中对应的元素会被高亮（得分曲线、基线、$\sigma$ 括号、精确梯度与估计的箭头、ES 的点云）。Tip: hover the highlighted symbols in the formulas; the matching elements in the figures light up (the score curve, the baseline, the $\sigma$ bracket, the exact and estimated gradient arrows, the ES cloud).

目标与对数导数技巧The objective and the log-derivative trick

沿用 20.1 节的记号：策略 $\pi_\theta(a\mid s)$，转移 $P(s'\mid s,a)$，初始分布 $\rho_0$，奖励 $r_{t+1}$（动作 $a_t$ 之后得到），折扣 $\gamma$。一条轨迹 $\tau=(s_0,a_0,s_1,a_1,\dots)$ 的概率和回报是

In the notation of Lesson 20.1: policy $\pi_\theta(a\mid s)$, transitions $P(s'\mid s,a)$, initial distribution $\rho_0$, rewards $r_{t+1}$ (received after action $a_t$), discount $\gamma$. A trajectory $\tau=(s_0,a_0,s_1,a_1,\dots)$ has probability and return

$$ p_\theta(\tau)=\rho_0(s_0)\prod_{t=0}^{T-1}\pi_\theta(a_t\mid s_t)\,P(s_{t+1}\mid s_t,a_t),\qquad G_0(\tau)=\sum_{t=0}^{T-1}\gamma^t r_{t+1},\qquad J(\theta)=\mathbb E_{\tau\sim p_\theta}\big[G_0(\tau)\big] $$

对 $J=\int p_\theta(\tau)G_0(\tau)\,d\tau$ 求导，$G_0$ 不依赖 $\theta$，导数全落在概率上。用恒等式 $\nabla p=p\,\nabla\log p$（对数导数技巧）把它变回一个期望：

Differentiate $J=\int p_\theta(\tau)G_0(\tau)\,d\tau$: $G_0$ does not depend on $\theta$, so the derivative falls entirely on the probability. The identity $\nabla p=p\,\nabla\log p$ (the log-derivative trick) turns it back into an expectation:

$$ \hl{grad}{\nabla_\theta J}=\int \nabla_\theta p_\theta(\tau)\,G_0(\tau)\,d\tau=\mathbb E_{\tau\sim p_\theta}\big[G_0(\tau)\,\nabla_\theta\log p_\theta(\tau)\big],\qquad \nabla_\theta\log p_\theta(\tau)=\sum_{t=0}^{T-1}\nabla_\theta\log\pi_\theta(a_t\mid s_t) $$

第二个等式是整个方法的关键：$\log p_\theta(\tau)$ 是 $\log\rho_0+\sum_t\log\pi_\theta+\sum_t\log P$ 的和，而 $\rho_0$ 和 $P$ 与 $\theta$ 无关，求导后消失。所以梯度里没有动力学：只需要能对自己的策略求 $\log$ 概率的导数。用 $N$ 条采样轨迹的平均代替期望，就是 REINFORCE：

The second equality is the key to the whole method: $\log p_\theta(\tau)$ is the sum $\log\rho_0+\sum_t\log\pi_\theta+\sum_t\log P$, and $\rho_0$ and $P$ do not depend on $\theta$, so they vanish under the derivative. The dynamics are not in the gradient: all you need is the derivative of the log-probability of your own policy. Replacing the expectation with an average over $N$ sampled trajectories gives REINFORCE:

$$ \hl{ghat}{\hat g}=\frac1N\sum_{i=1}^N\sum_{t=0}^{T-1}\nabla_\theta\log\pi_\theta\big(a^i_t\mid s^i_t\big)\,\Psi^i_t,\qquad \Psi_t\in\Big\{\,G_0,\ \ \hl{rtg}{G_t},\ \ G_t-\hl{base}{b_t},\ \ G_t-V^\pi(s_t)\,\Big\},\qquad G_t=\sum_{t'=t}^{T-1}\gamma^{t'-t}r_{t'+1} $$

四种 $\Psi_t$ 就是 FIG 3 的四级阶梯，期望都等于 $\nabla J$（$\gamma<1$ 时还要乘 $\gamma^t$，见误区 5）。FIG 2 到 FIG 4 用 $\gamma=1$。

The four choices of $\Psi_t$ are the four rungs of FIG 3, and all have expectation $\nabla J$ (with $\gamma<1$ there is an extra factor $\gamma^t$, see Pitfall 5). FIGS 2 to 4 use $\gamma=1$.

高斯策略的得分The score of a Gaussian policy

连续动作最常用的是高斯策略：网络（或任意可导函数）给出均值 $\mu_\theta(s)$，标准差 $\sigma$ 用一个独立参数 $\log\sigma$ 表示（取对数保证 $\sigma>0$，也让步长与尺度无关）。$\log\pi=-\tfrac{(a-\mu)^2}{2\sigma^2}-\log\sigma-\tfrac12\log2\pi$，于是

For continuous actions the usual choice is a Gaussian policy: a network (or any differentiable function) outputs the mean $\mu_\theta(s)$, and the standard deviation is a separate parameter $\log\sigma$ (the log keeps $\sigma>0$ and makes the step size scale-free). With $\log\pi=-\tfrac{(a-\mu)^2}{2\sigma^2}-\log\sigma-\tfrac12\log2\pi$,

$$ \hl{score}{\frac{\partial\log\pi}{\partial\mu}=\frac{a-\mu}{\sigma^2}}=\frac{\epsilon}{\hl{sigma}{\sigma}},\qquad \frac{\partial\log\pi}{\partial\log\sigma}=\frac{(a-\mu)^2}{\sigma^2}-1=\epsilon^2-1,\qquad \nabla_\theta\log\pi=\frac{a-\mu_\theta(s)}{\sigma^2}\,\nabla_\theta\mu_\theta(s) $$

其中 $\epsilon=(a-\mu)/\sigma$ 是这一步实际抽到的标准噪声。对 FIG 2 的 PD 策略，$\mu=-k_pe-k_d\dot q$，所以 $\nabla_\theta\mu=(-e,-\dot q)$：某一步的噪声 $\epsilon_t$ 推了关节一下，如果之后的回报好，$k_p$ 就朝 $-\epsilon_te_t$ 的方向调。对神经网络，$\nabla_\theta\mu$ 由反向传播（16.3 节）给出：把 $(a-\mu)/\sigma^2$ 乘以权重作为输出端的梯度回传就行。最后一个式子也说明了 $\sigma\to0$ 时为什么不能直接用：得分按 $1/\sigma$ 放大，估计的方差发散。确定性策略的梯度要换一条路，用 critic 对动作的导数 $\nabla_aQ$（DDPG/TD3，20.4 节）。

实现时从不手写这些梯度，而是构造一个“伪损失”，它的梯度恰好是 $-\hat g$，交给自动求导（FIG 5 的代码就是这样，只是手写了反向传播）：

Here $\epsilon=(a-\mu)/\sigma$ is the standard noise actually drawn at that step. For the PD policy of FIG 2, $\mu=-k_pe-k_d\dot q$, so $\nabla_\theta\mu=(-e,-\dot q)$: the noise $\epsilon_t$ gave the joint a nudge at some step, and if the return after it was good, $k_p$ moves in the direction of $-\epsilon_te_t$. For a neural network, $\nabla_\theta\mu$ comes from backpropagation (Lesson 16.3): feed $(a-\mu)/\sigma^2$ times the weight in as the gradient at the output. The last formula also shows why $\sigma\to0$ cannot be used directly: the score grows like $1/\sigma$ and the variance of the estimate diverges. The gradient of a deterministic policy needs another route, the derivative of a critic with respect to the action, $\nabla_aQ$ (DDPG/TD3, Lesson 20.4).

Implementations never write these gradients by hand. They build a "pseudo-loss" whose gradient is exactly $-\hat g$ and hand it to automatic differentiation (FIG 5's code does exactly this, only with the backward pass written out):

# 一次 REINFORCE 更新（PyTorch 风格的伪代码）
batch = [rollout(policy) for _ in range(N)]          # N 条轨迹: s_t, a_t, r_{t+1}
G     = reward_to_go(batch.r)                         # G_t = r_{t+1} + r_{t+2} + ...
adv   = G - baseline(batch)                           # 基线: 同一时刻其余轨迹的平均，或学到的 V(s_t)
adv   = adv / (adv.std() + 1e-8)                      # 归一化（可选）
logp  = policy.dist(batch.s).log_prob(batch.a)        # 高斯: -(a-mu)^2/(2 sigma^2) - log sigma
loss  = -(logp * adv.detach()).mean()                 # 伪损失：它的梯度是 -g_hat
opt.zero_grad(); loss.backward(); opt.step()          # 用过的样本随即丢弃（on-policy）

# one REINFORCE update (PyTorch-style pseudo-code)
batch = [rollout(policy) for _ in range(N)]          # N trajectories: s_t, a_t, r_{t+1}
G     = reward_to_go(batch.r)                         # G_t = r_{t+1} + r_{t+2} + ...
adv   = G - baseline(batch)                           # baseline: mean of the other trajectories at t, or a learned V(s_t)
adv   = adv / (adv.std() + 1e-8)                      # normalize (optional)
logp  = policy.dist(batch.s).log_prob(batch.a)        # Gaussian: -(a-mu)^2/(2 sigma^2) - log sigma
loss  = -(logp * adv.detach()).mean()                 # pseudo-loss: its gradient is -g_hat
opt.zero_grad(); loss.backward(); opt.step()          # the samples are then thrown away (on-policy)

为什么可以只用 reward-to-go，为什么基线不引入偏差Why reward-to-go is allowed, and why a baseline adds no bias

两件事都来自同一个事实：得分的期望为零。对任意状态 $s$，

Both follow from one fact: the score has zero mean. For any state $s$,

$$ \mathbb E_{a\sim\pi_\theta(\cdot\mid s)}\big[\nabla_\theta\log\pi_\theta(a\mid s)\big]=\int\nabla_\theta\pi_\theta(a\mid s)\,da=\nabla_\theta\!\int\pi_\theta(a\mid s)\,da=\nabla_\theta1=0 $$

因果性：对 $t'<t$，奖励 $r_{t'+1}$ 在第 $t$ 步的动作之前就已确定。先对 $a_t$ 求条件期望，$r_{t'+1}$ 是常数，可以提出来，剩下的得分期望为零，所以 $\mathbb E[r_{t'+1}\nabla\log\pi(a_t\mid s_t)]=0$。从 $G_0$ 里删掉这些项，期望不变，方差变小：这就是 reward-to-go。基线：任何只依赖 $s_t$（不依赖 $a_t$ 及之后）的 $b(s_t)$ 同理，$\mathbb E[b(s_t)\nabla\log\pi(a_t\mid s_t)]=0$。FIG 1 的“留一法”基线只用其他样本，与本样本的动作独立，所以严格无偏；如果用包括自己在内的批平均，会有一个 $1/N$ 量级的偏差。基线依赖本步动作就不行了：那会改变期望。

优势与策略梯度定理：给定 $(s_t,a_t)$，$G_t$ 的条件期望就是 $Q^\pi(s_t,a_t)$，所以 $G_t$ 可以换成 $Q^\pi$，基线取 $V^\pi$，得到

Causality: for $t'<t$ the reward $r_{t'+1}$ is settled before the action at step $t$. Take the conditional expectation over $a_t$ first: $r_{t'+1}$ is a constant, comes outside, and the remaining score has zero mean, so $\mathbb E[r_{t'+1}\nabla\log\pi(a_t\mid s_t)]=0$. Dropping these terms from $G_0$ leaves the expectation unchanged and lowers the variance: that is reward-to-go. Baselines: any $b(s_t)$ that depends only on $s_t$ (not on $a_t$ or anything after it) works the same way, $\mathbb E[b(s_t)\nabla\log\pi(a_t\mid s_t)]=0$. The "leave-one-out" baseline of FIG 1 uses only the other samples, which are independent of this sample's action, so it is exactly unbiased; the batch mean including the sample itself would add a bias of order $1/N$. A baseline that depends on this step's action is not allowed: it would change the expectation.

The advantage and the policy gradient theorem: given $(s_t,a_t)$, the conditional expectation of $G_t$ is $Q^\pi(s_t,a_t)$, so $G_t$ may be replaced by $Q^\pi$ and the baseline taken as $V^\pi$, giving

$$ \nabla_\theta J=\mathbb E\Big[\sum_{t}\gamma^t\,\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,A^\pi(s_t,a_t)\Big],\qquad A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),\qquad \mathbb E_{a\sim\pi}\big[A^\pi(s,a)\big]=0 $$

这是 Sutton 等人（1999）的策略梯度定理的轨迹形式。优势的均值为零：它只回答“这个动作比这个状态下的平均好多少”，把状态本身好坏带来的方差全部去掉。实际中 $V^\pi$ 未知，用一个网络去学它就是 actor-critic；用 TD 误差 $r_{t+1}+\gamma V(s_{t+1})-V(s_t)$ 代替 $G_t-V(s_t)$，方差更小但有偏，GAE 在两者之间插值（20.4 节）。

最优常数基线：在所有常数 $b$ 里，使方差最小的是 $b^\star=\mathbb E[\|\nabla\log p_\theta\|^2G_0]/\mathbb E[\|\nabla\log p_\theta\|^2]$，即按得分平方加权的平均回报（Greensmith 等，2004）。平均回报本身是一个简单且常常够好的近似。

This is the trajectory form of the policy gradient theorem of Sutton et al. (1999). The advantage has zero mean: it only answers "how much better is this action than the average in this state", which removes all the variance caused by how good the state itself is. In practice $V^\pi$ is unknown; learning it with a network is actor-critic, and replacing $G_t-V(s_t)$ with the TD error $r_{t+1}+\gamma V(s_{t+1})-V(s_t)$ gives lower variance but some bias, with GAE interpolating between the two (Lesson 20.4).

The optimal constant baseline: among constants $b$, the one with the least variance is $b^\star=\mathbb E[\|\nabla\log p_\theta\|^2G_0]/\mathbb E[\|\nabla\log p_\theta\|^2]$, the average return weighted by the squared score (Greensmith et al., 2004). The plain average return is a simple and often good-enough approximation.

方差的来源，以及 ES 的公式Where the variance comes from, and the ES formula

一条轨迹的 $\hat g$ 是 $T$ 项之和，每项是一个均值为零、大小约 $1/\sigma$ 的得分乘以一个回报。没有好的基线时，每一项都把整条轨迹的随机性（起点、其他步的噪声）带进来，方差大致随 $T$ 增长；控制频率翻倍，$T$ 翻倍，每一步噪声对回报的影响却减半，信噪比变差（FIG 4）。ES 的估计是

For one trajectory, $\hat g$ is a sum of $T$ terms, each a zero-mean score of size about $1/\sigma$ times a return. Without a good baseline each term brings in the randomness of the whole trajectory (the start, the noise at other steps), and the variance grows roughly with $T$; doubling the control rate doubles $T$ while halving each step's influence on the return, so the signal-to-noise ratio gets worse (FIG 4). The ES estimate is

$$ \hat g_{\hl{es}{\mathrm{ES}}}=\frac{1}{2M\sigma_\theta}\sum_{j=1}^M\Big[G\big(\theta+\sigma_\theta\boldsymbol\epsilon_j\big)-G\big(\theta-\sigma_\theta\boldsymbol\epsilon_j\big)\Big]\boldsymbol\epsilon_j,\qquad \boldsymbol\epsilon_j\sim\mathcal N(0,I_d) $$

它估计的是平滑后的 $J_{\sigma_\theta}(\theta)=\mathbb E[J(\theta+\sigma_\theta\boldsymbol\epsilon)]$ 的梯度（6.11 节），每对扰动只看到 $\boldsymbol\epsilon_j$ 方向上的一个差分，要拼出 $d$ 维的梯度至少需要 $d$ 量级的样本：方差随 $d$ 线性增长，与 $T$ 无关。所以二者各有地盘：参数少（几十维的步态参数、线性策略）而控制频率高时，ES 和随机搜索非常有竞争力（Mania 等，2018）；参数是百万维的神经网络时，在动作空间探索、配上学到的 critic 的策略梯度通常更好。Salimans 等（2017）用大量并行 worker 的 ES 训练过神经网络策略，靠的是并行规模。

It estimates the gradient of the smoothed $J_{\sigma_\theta}(\theta)=\mathbb E[J(\theta+\sigma_\theta\boldsymbol\epsilon)]$ (Lesson 6.11). Each pair of perturbations sees only one difference along $\boldsymbol\epsilon_j$, so assembling a $d$-dimensional gradient takes on the order of $d$ samples: the variance grows linearly with $d$ and does not depend on $T$. So each has its territory: with few parameters (gait parameters, linear policies) and a high control rate, ES and random search are very competitive (Mania et al., 2018); for a neural network with millions of parameters, exploring in action space with a learned critic is usually better. Salimans et al. (2017) did train neural-network policies with ES, relying on a very large number of parallel workers.

步长：为什么还需要 20.4 节The step size: why Lesson 20.4 is still needed

REINFORCE 是 on-policy 的：$\hat g$ 是当前策略下的期望，更新一次之后，旧样本就不再来自当前策略，只能丢掉（或者用重要性权重 $\pi_{\theta'}/\pi_{\theta}$ 修正，4.9 节）。而 $\theta$ 空间里的“一步”在策略空间里可能很大，也可能很小，取决于参数化：FIG 2 里 $\alpha$ 稍大，路径就开始在山脊上大幅跳动。自然梯度（Kakade 2001）用策略分布之间的 KL 散度（4.11 节）来度量步长，TRPO 把它变成约束，PPO 用裁剪的概率比近似它：这些都在 20.4 节。

REINFORCE is on-policy: $\hat g$ is an expectation under the current policy, so after one update the old samples no longer come from the current policy and must be thrown away (or corrected with importance weights $\pi_{\theta'}/\pi_{\theta}$, Lesson 4.9). And a "step" in $\theta$ space may be large or small in policy space, depending on the parametrization: in FIG 2, a slightly larger $\alpha$ already makes the path jump around on the ridge. The natural gradient (Kakade 2001) measures the step by the KL divergence between policy distributions (Lesson 4.11), TRPO turns this into a constraint, and PPO approximates it with a clipped probability ratio: all in Lesson 20.4.

6机器人中的它：一个神经网络策略控制受重力的机械臂In robotics: a neural-network policy for an arm under gravity

把前面的东西放到一个非线性任务上。一根竖直平面里的连杆（单位惯量）从自然下垂开始，要在 3 秒内转到每次随机给出的目标角 $q^\star\in[-2.4,2.4]$ rad 并停住：$\ddot q=\tau-5\sin q-0.5\,\dot q$，电机力矩被饱和在 $|\tau|\le8$（策略并不知道）。目标越高，需要的重力补偿 $5\sin q^\star$ 越大，而且它随 $q^\star$ 非线性变化，接近水平时最大。策略是 $\mu_\theta(s)=4\,f_\theta(e,\dot q/3,q^\star/2)$，$f_\theta$ 是一个两层各 24 个 tanh 单元的 MLP，加上一个可学习的 $\log\sigma$，初始 $\sigma=2$。每次更新抽 $N$ 条轨迹，优势是 reward-to-go 减去时间基线、再标准化（减去均值、除以标准差），网络用 Adam 更新：这就是第 5 节那段伪代码，只是反向传播是手写的（nn.js）。

Now put everything on a nonlinear task. A link in a vertical plane (unit inertia) starts hanging down and must turn to a target angle $q^\star\in[-2.4,2.4]$ rad, drawn at random each episode, and stop there within 3 seconds: $\ddot q=\tau-5\sin q-0.5\,\dot q$, with the motor torque saturated at $|\tau|\le8$ (the policy does not know this). The higher the target, the larger the gravity compensation $5\sin q^\star$ it needs, and that varies nonlinearly with $q^\star$, peaking at horizontal. The policy is $\mu_\theta(s)=4\,f_\theta(e,\dot q/3,q^\star/2)$, where $f_\theta$ is an MLP with two layers of 24 tanh units, plus a learnable $\log\sigma$, starting at $\sigma=2$. Each update draws $N$ trajectories; the advantage is the reward-to-go minus the time baseline, then standardized (mean subtracted, divided by the standard deviation), and the network is updated with Adam: the pseudo-code of Section 5, with the backward pass written out by hand (nn.js).

FIG 5实时训练：高斯神经网络策略学会把受重力的连杆举到任意角度Live training: a Gaussian neural-network policy learns to hold a link against gravity at any angle

左：琥珀色连杆按当前策略的均值（不加噪声）实时运动，从下垂开始、3 秒后重来；紫色虚线是目标角，拖动它末端的手柄可以换目标；细的琥珀线是末端走过的轨迹。左下的数字是这一次 3 秒结束时的误差。右上：测试误差，即 8 个固定目标（±0.4、±1、±1.6、±2.2 rad）上均值策略 3 秒末的平均 $|q-q^\star|$（rad，对数刻度），横轴是用过的轨迹条数；绿虚线是 0.1 rad。右下：$\sigma$ 的变化（紫）。每帧做几次更新，只要不超过约 7 ms（至少一次；$N=64$ 时一次更新就会超过），随机数有固定种子，同一种子每次训练结果相同。 Left: the amber link moves in real time under the mean of the current policy (no noise), starting from hanging down and restarting every 3 seconds; the purple dashed line is the target angle, and dragging the handle at its end changes the target; the thin amber line is the path of the tip. The number at bottom left is the error at the end of the current 3 seconds. Top right: the test error, the mean final $|q-q^\star|$ of the mean policy over 8 fixed targets (±0.4, ±1, ±1.6, ±2.2 rad), in rad on a log scale, against the number of trajectories used; the green dashed line is 0.1 rad. Bottom right: $\sigma$ over training (purple). Each frame runs as many updates as fit in about 7 ms (at least one; at $N=64$ a single update already takes longer), and the random numbers are seeded, so the same seed always trains the same way.

按“训练”：一开始连杆乱晃（测试误差约 1 rad 量级），几千条轨迹之后测试误差降到 0.1 rad 以下。注意曲线不是单调的：一次运气不好的更新可以让策略暂时变差。Press "Train": at first the link flails (test error of order 1 rad), and after a few thousand trajectories the test error drops below 0.1 rad. Notice that the curve is not monotonic: one unlucky update can make the policy worse for a while.
训练之后，把目标拖到 $|q^\star|\ge2$ rad（越过水平、举到转轴上方，需要很大的重力补偿）：均值策略 3 秒末的误差通常小于 0.15 rad（这个误差随每次更新漂移；如果还不够小，保持目标继续训练一会儿，或试试另一侧的对称目标）。网络从没被告诉过 $5\sin q$ 这一项，它是从回报里自己学出来的。After training, drag the target to $|q^\star|\ge2$ rad (past horizontal, above the pivot, which needs a large gravity compensation): the mean policy usually ends within 0.15 rad of it (this error drifts from update to update; if it is not small enough yet, keep the target and train a little longer, or try the mirror-image target on the other side). Nobody told the network about the $5\sin q$ term; it learned it from the return alone.

和 FIG 2 相比有几处变化，都是实际代码里的标准做法：优势除以标准差，学习率因此与奖励的尺度无关；$\log\sigma$ 是与状态无关的独立参数；动作超出电机范围时由环境截断，而 $\log\pi$ 按截断前的动作计算，这样估计仍然无偏（被截断的部分对回报的影响照样被正确地记账）。你也会注意到 $\sigma$ 降得很慢：这个任务里力矩噪声的代价很小，梯度几乎不催 $\sigma$ 变小。

Compared with FIG 2 a few things changed, all of them standard in real code: the advantages are divided by their standard deviation, so the learning rate does not depend on the scale of the reward; $\log\sigma$ is a separate, state-independent parameter; when an action exceeds the motor's range the environment clips it, while $\log\pi$ is computed for the action before clipping, which keeps the estimate unbiased (the effect of the clipping on the return is still booked correctly). You will also notice that $\sigma$ falls slowly: in this task torque noise costs little, so the gradient hardly pushes $\sigma$ down.

真实系统里In real systems

早期的真机策略搜索：Kohl 和 Stone（ICRA 2004）让 Sony AIBO 机器狗自己在场地上计时行走，通过扰动一个参数化小跑步态的参数来估计梯度，学出了比当时手调步态更快的步态，这在本质上就是参数空间的探索（FIG 4 的 ES 一侧）。Tedrake、Zhang 和 Seung（IROS 2004）在一台简单的三维双足机器人上用随机策略梯度在线学习行走，没有用模型。 运动基元：Peters 和 Schaal（2008）把策略梯度与自然梯度用在运动基元的参数上，在拟人机械臂上学习击球这类动作；Kober 和 Peters（2008）在 Barrett WAM 上从人的示教出发学会了“杯中球”。参数维数小、从模仿出发，正是策略梯度在真机上最能发挥作用的条件。现在：腿式机器人的运动控制大多用 PPO（20.4 节）在 Isaac Gym/Lab 这类大规模并行仿真里训练（23.1 节），PPO 的核心仍是本节的 $\nabla\log\pi\cdot A$，加上学到的 critic 和受控的步长。大语言模型的 RLHF 也用策略梯度：RLOO 就是本页的留一法基线（Kool 等，2019），DeepSeekMath 提出的 GRPO 用同一问题多次采样的组内均值做基线、再除以组内标准差，和 FIG 5 的做法如出一辙。

Early policy search on real robots: Kohl and Stone (ICRA 2004) had Sony AIBO robot dogs time their own walks across a field and estimated gradients by perturbing the parameters of a parametrized trot gait, learning a gait faster than the hand-tuned gaits of the time; in essence parameter-space exploration (the ES side of FIG 4). Tedrake, Zhang and Seung (IROS 2004) used a stochastic policy gradient to learn walking online on a simple 3D biped, without a model. Motor primitives: Peters and Schaal (2008) applied policy gradients and natural gradients to the parameters of motor primitives, learning skills such as hitting a ball on an anthropomorphic arm; Kober and Peters (2008) learned ball-in-a-cup on a Barrett WAM starting from a human demonstration. Few parameters and a start from imitation are exactly the conditions under which policy gradients work best on hardware. Today: legged locomotion is mostly trained with PPO (Lesson 20.4) in massively parallel simulators such as Isaac Gym/Lab (Lesson 23.1), and the core of PPO is still this lesson's $\nabla\log\pi\cdot A$, plus a learned critic and a controlled step size. RLHF for large language models uses policy gradients too: RLOO is this page's leave-one-out baseline (Kool et al., 2019), and the GRPO introduced with DeepSeekMath uses the mean over several samples for the same prompt as the baseline and divides by their standard deviation, just like FIG 5.

7常见误区Common pitfalls

误区 1：基线依赖了动作，或者干脆不用基线Pitfall 1: a baseline that depends on the action, or no baseline at all

基线只能依赖状态（以及与本样本动作独立的东西，比如其他轨迹）。用 $Q(s_t,a_t)$ 本身、或者包含本步之后奖励的量做“基线”，期望就变了，算法会稳定地朝错误方向走。反过来，不用基线虽然无偏，FIG 1 已经演示了后果：奖励整体平移一下方差就放大十倍，所有样本都成功时策略还会被随机地踢来踢去。

A baseline may depend only on the state (and on things independent of this sample's action, such as other trajectories). Using $Q(s_t,a_t)$ itself, or anything containing rewards from this step on, as a "baseline" changes the expectation, and the algorithm walks steadily in the wrong direction. Conversely, no baseline is unbiased but FIG 1 showed the consequences: shift all rewards and the variance grows tenfold; once every sample succeeds, the policy still gets kicked around at random.

误区 2：重复使用旧样本，或步长太大Pitfall 2: reusing old samples, or steps that are too large

REINFORCE 的梯度是当前策略下的期望。把上一轮的轨迹存起来再用（像 DQN 的回放缓存那样），就是在用别的策略的样本估计这个策略的梯度，有偏；要复用必须乘重要性权重，而权重一大方差就爆炸，PPO 的裁剪就是为此设计的。步长同理：一次太大的更新会让策略跳到地形上很差的地方，之后采到的数据也随之变差，很难恢复。

The REINFORCE gradient is an expectation under the current policy. Storing last round's trajectories and using them again (like DQN's replay buffer) estimates this policy's gradient from another policy's samples, which is biased; reuse needs importance weights, whose variance explodes when they get large, and PPO's clipping was designed for exactly that. Steps are the same story: one update that is too large can throw the policy onto a bad part of the landscape, after which the data it collects is worse too, and recovery is hard.

误区 3：探索噪声过早消失Pitfall 3: exploration noise that vanishes too early

$\sigma$ 也是被梯度优化的，它会朝“当前附近最好”的方向收缩。FIG 1 的“两个峰”里策略缩到矮峰上，$\sigma$ 一小，远处的高峰就再也看不见了，梯度为零，学习停止。常见对策：给 $\log\sigma$ 设下限、在目标里加一项熵奖励 $\beta\,\mathcal H(\pi)$（对高斯就是 $\beta\log\sigma$），或者干脆把熵作为目标的一部分，后者就是 20.4 节的 SAC。也别走到另一头：$\sigma$ 太大，动作大多被电机饱和截断，信号反而少了。

$\sigma$ is optimized by the gradient too, and it shrinks towards "the best thing nearby". In FIG 1's "two peaks" the policy shrinks onto the short peak; once $\sigma$ is small the tall peak is out of sight, the gradient is zero and learning stops. Common remedies: a floor on $\log\sigma$, an entropy bonus $\beta\,\mathcal H(\pi)$ in the objective (for a Gaussian that is $\beta\log\sigma$), or making entropy part of the objective outright, which is the SAC of Lesson 20.4. Do not go to the other extreme either: with $\sigma$ too large most actions are clipped by the motor limits, and the signal gets weaker, not stronger.

误区 4：动作有界时 log 概率算错Pitfall 4: getting the log-probability wrong when actions are bounded

电机力矩、关节速度都有上下限。两种正确做法：像 FIG 5 那样从无界高斯采样，由环境截断，$\log\pi$ 按截断前的样本算；或者把样本过一个 $\tanh$ 压进范围，这时 $\log\pi$ 必须加上变量替换的雅可比修正 $-\sum\log(1-\tanh^2u)$（19.3 节的变量替换，SAC 就这样做）。错误做法是按截断后的动作去算未截断高斯的 $\log\pi$：边界上的大量样本会得到错误的得分。

Motor torques and joint speeds have limits. Two correct options: sample from an unbounded Gaussian and let the environment clip it, as in FIG 5, computing $\log\pi$ for the sample before clipping; or squash the sample into range with $\tanh$, in which case $\log\pi$ needs the change-of-variables Jacobian correction $-\sum\log(1-\tanh^2u)$ (the change of variables of Lesson 19.3; SAC does exactly this). The wrong option is to evaluate the unclipped Gaussian's $\log\pi$ at the action after clipping: the many samples sitting on the boundary get wrong scores.

误区 5：折扣用得不一致，或拿回报曲线当唯一证据Pitfall 5: inconsistent discounting, or the return curve as the only evidence

对折扣目标 $\mathbb E[\sum\gamma^tr_{t+1}]$，严格的梯度在第 $t$ 项上还有一个因子 $\gamma^t$。几乎所有实现都把它省掉、只在 $G_t$ 里用 $\gamma$：这时算出来的已经不是任何目标的精确梯度，$\gamma$ 实际上成了一个“只看未来多远”的降方差旋钮。这通常没问题，但要清楚自己优化的是什么，比较不同 $\gamma$ 时要用同一个未打折的指标。另外，学习曲线本身就很吵（FIG 5），只看一个随机种子的一条曲线下结论很危险：至少比较几个种子，并在固定的测试状态上评估均值策略。

For the discounted objective $\mathbb E[\sum\gamma^tr_{t+1}]$, the exact gradient carries an extra factor $\gamma^t$ on the term for step $t$. Almost all implementations drop it and use $\gamma$ only inside $G_t$: what they compute is then no longer the exact gradient of any objective, and $\gamma$ effectively becomes a variance-reduction knob for "how far ahead to look". That is usually fine, but know what you are optimizing, and compare different $\gamma$ on the same undiscounted metric. Also, the learning curve itself is noisy (FIG 5), and drawing conclusions from one curve of one seed is risky: compare several seeds, and evaluate the mean policy on fixed test states.

8小练习Check yourself

1. REINFORCE 的梯度估计为什么不需要环境的动力学模型？1. Why does the REINFORCE gradient estimate not need a model of the environment's dynamics?

选项 B。轨迹概率是初始分布、各步策略和各步转移的乘积，取对数变成和；只有策略那几项含 $\theta$。动力学只通过“采样出来的轨迹”间接参与，不需要写出它，也不需要它可导。选项 A 不对：FIG 5 的机械臂就是非线性的；FIG 2 的线性只是为了让页面能画出精确地形。选项 C、D 描述的是别的方法（有限差分、基于模型或价值的方法）。Option B. The trajectory probability is a product of the initial distribution, the policy at each step and the transition at each step; its log is a sum, and only the policy terms contain $\theta$. The dynamics enter only through the sampled trajectories, so they never need to be written down or differentiated. Option A is wrong: FIG 5's arm is nonlinear, and FIG 2 is linear only so that the page can draw the exact landscape. Options C and D describe other methods (finite differences, model- or value-based methods).

2. 不用基线，把每一步的奖励都加上同一个常数 5（最优策略不变）。REINFORCE 估计会怎样？2. Without a baseline, add the same constant 5 to every reward (the optimal policy does not change). What happens to the REINFORCE estimate?

选项 C。常数乘以得分的期望是零（得分均值为零），所以期望不变；但每个样本都多了一项 $5\nabla\log\pi$，它们只是在零附近乱跳，方差变大。FIG 1 里 $c=5$ 时离散程度是 $c=0$ 时的十倍左右。基线正是用来减掉这种与动作无关的常数部分的：开了留一法基线，估计与 $c$ 完全无关。Option C. A constant times the score has zero expectation (the score has zero mean), so the expectation is unchanged; but each sample gets an extra term $5\nabla\log\pi$ that only jitters around zero, so the variance grows. In FIG 1 the spread at $c=5$ is about ten times the spread at $c=0$. That action-independent constant part is exactly what a baseline subtracts: with the leave-one-out baseline the estimate does not depend on $c$ at all.

3. 下面哪一个可以用作基线 $b$，而不让 $\Psi_t=G_t-b$ 的估计产生偏差？3. Which of these can serve as a baseline $b$ without making the estimate with $\Psi_t=G_t-b$ biased?

选项 A。无偏的条件是 $\mathbb E[b\,\nabla\log\pi(a_t\mid s_t)]=0$，只要 $b$ 在给定 $s_t$ 后与 $a_t$ 无关，它就能从对 $a_t$ 的期望里提出来，乘上均值为零的得分。$V$ 学得准不准只影响方差，不影响偏差。B、C、D 都依赖 $a_t$（$G_t$ 和 $Q(s_t,a_t)$ 都随 $a_t$ 变），会改变期望；C 更是把信号整个减没了。Option A. Unbiasedness needs $\mathbb E[b\,\nabla\log\pi(a_t\mid s_t)]=0$; as long as $b$ does not depend on $a_t$ given $s_t$, it comes out of the expectation over $a_t$ and multiplies a zero-mean score. How accurate $V$ is affects only the variance, not the bias. B, C and D all depend on $a_t$ ($G_t$ and $Q(s_t,a_t)$ both change with $a_t$) and change the expectation; C even subtracts the whole signal away.

4. 想得到一个确定性策略，能不能直接把高斯策略的 $\sigma$ 设成接近 0 再用 REINFORCE？4. To get a deterministic policy, can you just set the Gaussian policy's $\sigma$ close to 0 and keep using REINFORCE?

选项 D。$\sigma$ 越小，每个样本的动作离均值越近，回报之间的差别越小，而得分却越大：信号缩小、噪声放大。FIG 3 里 B 点 $\sigma$ 从 0.5 降到 0.2，“$G_t-b_t$”的相对误差从 3.0 涨到 16.5。确定性策略本身没有问题，$J$ 也存在（排除 C），只是它的梯度要靠 $\nabla_aQ(s,a)$ 链式传给参数，这就需要一个可导的 critic（20.4 节）。Option D. The smaller $\sigma$, the closer each sampled action is to the mean and the smaller the differences in return, while the score grows: the signal shrinks and the noise grows. In FIG 3 at B, lowering $\sigma$ from 0.5 to 0.2 raises the relative error of "$G_t-b_t$" from 3.0 to 16.5. A deterministic policy is perfectly fine and $J$ exists (not C); its gradient just has to be chained through $\nabla_aQ(s,a)$ into the parameters, which needs a differentiable critic (Lesson 20.4).

5. 一个只有 12 个参数的步态控制器，控制频率 500 Hz、每条轨迹 10 秒。在没有 critic 的情况下，哪种估计更可能有效？5. A gait controller with only 12 parameters runs at 500 Hz and each trajectory lasts 10 seconds. Without a critic, which estimator is more likely to work well?

选项 B。每条轨迹有 5000 步，动作噪声每一步都在注入，单步噪声对回报的影响极小，不靠 critic 很难把它从回报的随机性里分辨出来（FIG 4：频率从 10 Hz 升到 50 Hz，余弦已从约 0.61 掉到约 0.35）。而 12 维参数空间里的扰动与步数无关。Kohl 和 Stone 的 AIBO 步态学习正是这种情形。两者都源于对数导数技巧，但探索发生在不同空间，方差完全不同（排除 C）；DQN 需要离散动作（排除 D）。Option B. A trajectory has 5000 steps and action noise is injected at every one; the effect of a single step's noise on the return is tiny and, without a critic, very hard to tell apart from the randomness of the return (FIG 4: going from 10 Hz to 50 Hz already drops the cosine from about 0.61 to about 0.35). Perturbations in a 12-dimensional parameter space do not care about the step count. Kohl and Stone's AIBO gait learning was exactly this situation. Both come from the log-derivative trick, but they explore in different spaces and their variances are entirely different (not C); DQN needs discrete actions (not D).

9延伸阅读Further reading

R. J. Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, Machine Learning 8, 1992 — REINFORCE 的原始论文，包括基线the original REINFORCE paper, baselines included
R. S. Sutton, D. McAllester, S. Singh & Y. Mansour, Policy Gradient Methods for Reinforcement Learning with Function Approximation, NIPS 1999 — 策略梯度定理与相容的函数近似the policy gradient theorem and compatible function approximation
R. S. Sutton & A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018 — 第 13 章：策略梯度方法Chapter 13: policy gradient methods
E. Greensmith, P. L. Bartlett & J. Baxter, Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning, JMLR 2004 — 基线与最优基线的分析the analysis of baselines and the optimal baseline
S. Kakade, A Natural Policy Gradient, NIPS 2001 — 用策略分布的几何度量步长，TRPO 与 PPO 的前身measuring the step by the geometry of policy distributions, the ancestor of TRPO and PPO
J. Peters & S. Schaal, Reinforcement Learning of Motor Skills with Policy Gradients, Neural Networks 21(4), 2008 — 机器人运动技能上的策略梯度与自然 actor-critic 综述policy gradients and the natural actor-critic for robot motor skills
N. Kohl & P. Stone, Policy Gradient Reinforcement Learning for Fast Quadrupedal Locomotion, ICRA 2004 — 在真实 AIBO 上学步态learning a gait on real AIBOs
R. Tedrake, T. W. Zhang & H. S. Seung, Stochastic Policy Gradient Reinforcement Learning on a Simple 3D Biped, IROS 2004 — 在真机上在线学习双足行走learning biped walking online on the hardware
J. Kober & J. Peters, Policy Search for Motor Primitives in Robotics, NIPS 2008 — PoWER 与 Barrett WAM 上的杯中球PoWER and ball-in-a-cup on a Barrett WAM
M. P. Deisenroth, G. Neumann & J. Peters, A Survey on Policy Search for Robotics, Foundations and Trends in Robotics, 2013 — 机器人策略搜索的全景：策略梯度、EM 类方法、基于模型的方法the landscape of policy search for robots: policy gradients, EM-style and model-based methods
T. Salimans, J. Ho, X. Chen, S. Sidor & I. Sutskever, Evolution Strategies as a Scalable Alternative to Reinforcement Learning, arXiv:1703.03864, 2017 — 大规模并行的参数空间探索massively parallel parameter-space exploration
H. Mania, A. Guy & B. Recht, Simple Random Search of Static Linear Policies is Competitive for Reinforcement Learning, NeurIPS 2018 — 线性策略加随机搜索在连续控制基准上的表现random search over linear policies on continuous-control benchmarks
J. Schulman, P. Moritz, S. Levine, M. Jordan & P. Abbeel, High-Dimensional Continuous Control Using Generalized Advantage Estimation, ICLR 2016 — GAE：优势估计的偏差与方差折中（20.4 节）GAE: the bias-variance trade-off in advantage estimation (Lesson 20.4)
S. Mohamed, M. Rosca, M. Figurnov & A. Mnih, Monte Carlo Gradient Estimation in Machine Learning, JMLR 2020 — 得分函数、重参数化等梯度估计方法的系统综述a systematic review of score-function, pathwise and other gradient estimators
W. Kool, H. van Hoof & M. Welling, Buy 4 REINFORCE Samples, Get a Baseline for Free!, ICLR 2019 Workshop — 留一法基线（RLOO）the leave-one-out baseline (RLOO)