Part I · 第 4 章概率 · 4.11Part I · Ch. 4 Probability · 4.11

信息论入门A First Look at Information Theory

"不确定性"能不能量出来？熵就是答案：一个分布平均有多"意外"。从它出发，交叉熵和 KL 散度衡量"我的模型离真实分布有多远"，这正是分类网络的损失函数；互信息衡量"这个传感器平均能帮我消除多少不确定性"，这正是主动感知（选下一个观察位置）的依据。四个量，一套直觉。 Can "uncertainty" be measured? Entropy is the answer: how surprising a distribution is on average. From it, cross-entropy and the KL divergence measure "how far my model is from the true distribution", which is exactly the loss of a classification network; mutual information measures "how much uncertainty this sensor removes on average", which is the basis of active perception (choosing the next viewpoint). Four quantities, one intuition.

约 25 分钟~25 min 前置：4.1 随机变量与分布 · 4.2 条件概率与贝叶斯 · 4.6 最大似然（对照）Prereqs: 4.1 Random Variables · 4.2 Conditional Probability & Bayes · 4.6 MLE (for contrast) 5 个交互5 interactives

1为什么机器人需要它Why robots need this

信息论的几个量在机器人与机器学习里无处不在，却常常被当作"公式"背下来：

交叉熵损失：分类网络（识别物体、判断地形、行为克隆里的离散动作）几乎都用它训练。为什么是它？
KL 散度：PPO、TRPO 限制"新策略不要离旧策略太远"；VAE 的损失里有一项让隐变量分布靠近先验；SLAM 与滤波里比较两个高斯信念差多少。
熵：一个信念有多不确定？粒子滤波的 $N_{\mathrm{eff}}$、卡尔曼滤波协方差椭圆的"体积"，其实都在量熵；强化学习里"最大熵"让策略保持探索。
互信息与信息增益：机器人下一步该看哪里、走到哪里再测，才能最快消除对地图或自身位置的不确定性？答案是选期望信息增益最大的动作（4.7 节末尾已经用过了）。传感器放哪里、要不要加一个传感器，也是同一个问题。

这些量看似各管一摊，背后其实只有一个想法：用概率来度量"意外"。

A handful of information-theoretic quantities appear all over robotics and machine learning, and are often memorized as "formulas":

The cross-entropy loss: classification networks (recognizing objects, judging terrain, discrete actions in behavior cloning) are almost all trained with it. Why that one?
The KL divergence: PPO and TRPO limit "the new policy must not stray too far from the old one"; a VAE's loss has a term pulling the latent distribution toward the prior; SLAM and filtering compare how far apart two Gaussian beliefs are.
Entropy: how uncertain is a belief? The $N_{\mathrm{eff}}$ of a particle filter and the "volume" of a Kalman covariance ellipse are really measuring entropy; "maximum entropy" in reinforcement learning keeps a policy exploring.
Mutual information and information gain: where should the robot look or drive next to remove its uncertainty about the map or its own pose fastest? Choose the action with the largest expected information gain (already used at the end of Lesson 4.7). Where to put a sensor, and whether to add one at all, is the same question.

They look like separate topics, but behind them is a single idea: using probability to measure "surprise".

2直觉：意外程度，与"平均要问几个问题"Intuition: surprise, and "how many questions on average"

一个事件越不可能，发生时你越意外。把这个直觉写成数：事件概率为 $p$ 时，意外程度（自信息）是 $-\log_2 p$ 比特。必然发生（$p=1$）的事 0 比特；抛一枚硬币得到正面（$p=\tfrac12$）是 1 比特；$p=\tfrac18$ 是 3 比特。取对数是为了让独立事件的意外相加：两个独立事件同时发生，概率相乘，意外相加。

熵就是意外程度的平均值：$H(p)=-\sum_i p_i\log_2 p_i$。它还有一个很实在的含义：用"是 / 否"问题去猜一个按 $p$ 分布的结果，最聪明的问法平均要问大约 $H(p)$ 个问题。均匀分布最难猜（熵最大），确定的事不用问（熵为 0）。拖动下面的柱子，感受一下。

The less probable an event, the more surprised you are when it happens. Put numbers on that intuition: an event of probability $p$ carries surprise (self-information) of $-\log_2 p$ bits. A certain event ($p=1$) is 0 bits; heads on a fair coin ($p=\tfrac12$) is 1 bit; $p=\tfrac18$ is 3 bits. The logarithm is there to make the surprise of independent events add: two independent events together multiply their probabilities and add their surprise.

Entropy is the average surprise: $H(p)=-\sum_i p_i\log_2 p_i$. It also has a very practical meaning: to guess an outcome distributed as $p$ with yes/no questions, the cleverest strategy needs about $H(p)$ questions on average. A uniform distribution is hardest to guess (maximum entropy); a certain outcome needs no question (entropy 0). Drag the bars below and feel it.

FIG 1熵：平均意外程度Entropy: the average surprise

上图：6 种结果的概率 $p_i$，拖动柱顶改变，其余柱子会按比例调整使总和为 1。中图：每种结果的意外 $-\log_2 p_i$（封顶 8 比特，$p=0$ 时为无穷）。下条：熵 = 各结果的"概率 × 意外"依次叠起来，满条是 $\log_2 6\approx2.585$ 比特（均匀分布时的最大值）。注意稀有结果意外很大，但它被概率打了折扣，对熵的贡献并不大。 Top: probabilities $p_i$ of six outcomes. Drag a bar's top to change it; the other bars rescale proportionally so the total stays 1. Middle: the surprise $-\log_2 p_i$ of each outcome (capped at 8 bits; infinite at $p=0$). Bottom strip: entropy = the "probability × surprise" of each outcome stacked up; a full strip is $\log_2 6\approx2.585$ bits (the maximum, reached by the uniform distribution). Note that a rare outcome is very surprising but is discounted by its probability, so it contributes modestly to the entropy.

试一试Try this

让熵达到最大（超过 2.55 比特）。这个分布长什么样？Reach the maximum entropy (above 2.55 bits). What does the distribution look like?
让熵低于 0.5 比特，同时保证最小的那个结果概率不为 0。"几乎确定"长什么样？Get the entropy below 0.5 bits while no outcome has probability exactly 0. What does "almost certain" look like?
制造一个概率低于 0.03 的结果，读它的意外：超过 5 比特。再看它对熵的贡献有多小。Create an outcome with probability below 0.03 and read its surprise: over 5 bits. Then see how little it adds to the entropy.

核心洞见Key insight

熵衡量分布的"平坦程度"，而不是数值的大小：把结果的标签换一换，熵不变。均匀 = 最不确定，尖峰 = 最确定。机器人里，"信念的熵"就是它对自己状态的无知程度，这是 4.7 节 FIG 1 右侧读数里的那个"熵 H"。

Entropy measures how flat a distribution is, not the size of the values: relabel the outcomes and it does not change. Uniform = most uncertain, a sharp peak = most certain. In robotics the "entropy of the belief" is the robot's ignorance about its own state; it is the "entropy H" in the readout of Lesson 4.7's FIG 1.

3动手玩：我的模型离真相有多远？Play: how far is my model from the truth?

现在有两个分布：真实的 $p$（比如"这张图片里到底是哪一类"），和你的模型给出的 $q$（网络输出的 softmax）。如果按 $q$ 去设计"问题清单"（编码），而世界实际按 $p$ 发生，你平均要付出的比特数是交叉熵 $H(p,q)=-\sum_i p_i\log_2 q_i$。它总是不小于 $H(p)$；多出来的那部分，叫 KL 散度（"用错了模型而白白多付的比特数"）：

$$ \underbrace{H(p,q)}_{\hl{CE}{\text{cross-entropy}}} \;=\; \underbrace{H(p)}_{\hl{H}{\text{truth's entropy}}} \;+\; \underbrace{D_{\mathrm{KL}}(p\,\|\,q)}_{\hl{KL}{\text{penalty}}} $$ $$ D_{\mathrm{KL}}(p\,\|\,q)=\sum_i p_i\log_2\frac{p_i}{q_i}\ \ge\ 0 $$

拖动 $q$ 的柱顶，让它靠近 $p$，看那条"交叉熵"是怎样退到只剩 $H(p)$ 的。

Now there are two distributions: the true $p$ (say, "which class is really in this picture") and $q$, your model's output (a network's softmax). If you design your "question list" (code) around $q$ while the world actually follows $p$, the average number of bits you pay is the cross-entropy $H(p,q)=-\sum_i p_i\log_2 q_i$. It is never smaller than $H(p)$; the extra part is the KL divergence ("the bits wasted by using the wrong model"):

Drag the tops of $q$'s bars toward $p$ and watch the "cross-entropy" strip shrink back to just $H(p)$.

FIG 2交叉熵 = 熵 + KLCross-entropy = entropy + KL

黄色 = 真实分布 $p$，青色 = 你的模型 $q$（拖柱顶）。下面的长条是交叉熵，由两段叠成：黄段 = $H(p)$（不可避免的部分），红段 = KL（模型错造成的多余部分），满条 = 6 比特。每个 $q_i$ 最小取 0.0001，所以"把真值概率压到 0"的惩罚最多约 13 比特。 Amber = the true distribution $p$, cyan = your model $q$ (drag the tops). The strip below is the cross-entropy, stacked from two parts: the amber part = $H(p)$ (unavoidable) and the red part = KL (the excess caused by the wrong model); a full strip is 6 bits. Each $q_i$ is floored at 0.0001, so the penalty for "putting probability 0 on the truth" is at most about 13 bits.

试一试Try this

最小化损失：让 $q$ 贴近 $p$，KL 低于 0.02 比特。此时交叉熵等于多少？（等于 $H(p)$。）Minimize the loss: bring $q$ close to $p$ until KL is below 0.02 bits. What does the cross-entropy equal then? (It equals $H(p)$.)
分类损失：选"独热标签"，此时 $H(p)=0$，交叉熵就是 $-\log_2 q_{\text{正确类}}$。让 $q$ 对正确类的概率低于 0.05：损失超过 4.3 比特。The classification loss: pick "one-hot label"; now $H(p)=0$ and the cross-entropy is just $-\log_2 q_{\text{correct}}$. Push $q$'s probability of the correct class below 0.05: the loss exceeds 4.3 bits.
不对称：找一个 $q$，使 $D_{\mathrm{KL}}(p\|q)$ 与 $D_{\mathrm{KL}}(q\|p)$ 相差超过 0.5 比特。KL 不是距离。Asymmetry: find a $q$ for which $D_{\mathrm{KL}}(p\|q)$ and $D_{\mathrm{KL}}(q\|p)$ differ by more than 0.5 bits. KL is not a distance.

为什么分类用交叉熵Why classification uses cross-entropy

训练集是 $n$ 个样本，把它们的经验分布记作 $\hat p$。对数似然 $\frac1n\sum_i \log q(x_i)$ 恰好等于 $-H(\hat p,q)$。所以最大化似然 = 最小化交叉熵 = 最小化 $D_{\mathrm{KL}}(\hat p\|q)$（因为 $H(\hat p)$ 不依赖模型）。4.6 节的最大似然与这里的 KL 是同一件事的两种说法；平方误差对应的是高斯似然，交叉熵对应的是分类（类别分布）的似然。

A training set has $n$ samples; call their empirical distribution $\hat p$. The average log-likelihood $\frac1n\sum_i \log q(x_i)$ is exactly $-H(\hat p,q)$. Hence maximizing the likelihood = minimizing the cross-entropy = minimizing $D_{\mathrm{KL}}(\hat p\|q)$ (since $H(\hat p)$ does not depend on the model). The maximum likelihood of Lesson 4.6 and the KL here are the same thing in two languages; squared error corresponds to a Gaussian likelihood, cross-entropy to the likelihood of a categorical distribution (classification).

4第二个视角：KL 的两个方向，"覆盖"还是"抓峰"A second view: the two directions of KL, "cover" versus "seek a mode"

$D_{\mathrm{KL}}(p\|q)$ 与 $D_{\mathrm{KL}}(q\|p)$ 不相等，在连续分布里差别很有教育意义。用一个高斯 $q$ 去拟合一个双峰的 $p$：

前向 KL $D_{\mathrm{KL}}(p\|q)$：只要 $p$ 有质量的地方 $q$ 几乎为 0，就被重罚。所以 $q$ 会把所有的峰都罩住，摊得很宽（"覆盖质量"）。最小化它就是高斯的矩匹配，也是最大似然。
反向 KL $D_{\mathrm{KL}}(q\|p)$：$q$ 有质量的地方 $p$ 几乎为 0，才被重罚。于是 $q$ 老老实实地蹲在某一个峰上（"抓峰"），宁可丢掉另一个峰。变分推断常用这个方向。

下面两个按钮真的在 $(\mu,\sigma)$ 上暴力搜索最优高斯。下方曲线是两个 KL 的被积函数，面积就是 KL 的值。

$D_{\mathrm{KL}}(p\|q)$ and $D_{\mathrm{KL}}(q\|p)$ are not equal, and for continuous distributions the difference is instructive. Fit one Gaussian $q$ to a bimodal $p$:

Forward KL $D_{\mathrm{KL}}(p\|q)$: heavily penalizes any place where $p$ has mass but $q$ is nearly 0. So $q$ spreads wide enough to cover every peak ("mass covering"). Minimizing it is moment matching for a Gaussian, i.e. maximum likelihood.
Reverse KL $D_{\mathrm{KL}}(q\|p)$: heavily penalizes places where $q$ has mass but $p$ is nearly 0. So $q$ sits obediently on one peak ("mode seeking"), preferring to drop the other peak. Variational inference often uses this direction.

The two buttons below really do a brute-force search over $(\mu,\sigma)$ for the best Gaussian. The lower curves are the integrands of the two KLs; their areas are the KL values.

FIG 3一个高斯拟合双峰分布：前向与反向 KLOne Gaussian fitting a bimodal distribution: forward versus reverse KL

目标 p（黄）与高斯 q（青）TARGET p (AMBER) AND GAUSSIAN q (CYAN)

两个 KL 的被积函数INTEGRANDS OF THE TWO KLs

青色高斯可以拖动：山顶改均值 $\mu$，腰部改标准差 $\sigma$。下图两条曲线：蓝 = 前向被积函数 $p\log_2(p/q)$，绿 = 反向被积函数 $q\log_2(q/p)$；曲线下的面积 = KL。被积函数在个别点可以为负，但总面积不会为负。 Drag the cyan Gaussian: its peak sets the mean $\mu$, its flank the standard deviation $\sigma$. The two lower curves: blue = the forward integrand $p\log_2(p/q)$, green = the reverse integrand $q\log_2(q/p)$; the area under each curve is the KL. An integrand can be negative at individual points, but its total area is never negative.

试一试Try this

按"最优前向 KL"：高斯摊宽、盖住两个峰，中间那块没有 $p$ 的地方也有 $q$。Press "Best forward KL": the Gaussian spreads wide and covers both peaks, putting mass even in the gap where $p$ is small.
按"最优反向 KL"：高斯收紧、只贴住一个峰。比较两个 KL 值，哪个方向对"丢掉一个峰"更宽容？Press "Best reverse KL": the Gaussian tightens onto a single peak. Compare the two KL values: which direction is more forgiving of "dropping a peak"?
手动把高斯拖到前向 KL 在最优值的 1.05 倍以内。Drag the Gaussian by hand until the forward KL is within 1.05× of the optimum.

5第三个视角：互信息，传感器值多少钱？A third view: mutual information, what is a sensor worth?

一个门可能开着，也可能关着（先验 $\pi=P(\text{开})$）。机器人有个传感器，读数 $Z$ 说"开"或"关"，但不一定准：门开时说"开"的概率是真阳性率 $\mathrm{TPR}$，门关时误说"开"的概率是假阳性率 $\mathrm{FPR}$。

读数前你对门的不确定性是 $H(X)$；读到读数后平均剩下 $H(X\mid Z)$。互信息就是减少的那一部分：$I(X;Z)=H(X)-H(X\mid Z)$。它回答"这个传感器平均能提供多少比特的信息？"。

A door may be open or closed (prior $\pi=P(\text{open})$). The robot has a sensor whose reading $Z$ says "open" or "closed", though not always correctly: with the door open it says "open" with the true-positive rate $\mathrm{TPR}$; with the door closed it wrongly says "open" with the false-positive rate $\mathrm{FPR}$.

Before reading, your uncertainty about the door is $H(X)$; after the reading, on average $H(X\mid Z)$ is left. The mutual information is the part that disappeared: $I(X;Z)=H(X)-H(X\mid Z)$. It answers "how many bits of information does this sensor provide on average?"

FIG 4互信息：传感器的价值Mutual information: the value of a sensor

上方"马赛克"：横向分成"门开 / 门关"两列（宽度 = 先验），每列再按读数分成两格，每格面积 = 联合概率 $p(x,z)$。下方长条：总长 = $H(X)$，绿段 = 互信息 $I(X;Z)$（读数带来的），黄段 = $H(X\mid Z)$（读数后还剩的）。若读数和门互相独立（TPR = FPR），所有列里格子的高度一样，互信息为 0。 The "mosaic" above: split horizontally into "open / closed" columns (width = prior); each column is split by the reading, so each cell's area is the joint probability $p(x,z)$. The bar below: total length = $H(X)$, the green part = mutual information $I(X;Z)$ (what the reading delivers), the amber part = $H(X\mid Z)$ (what remains after reading). If the reading and the door are independent (TPR = FPR), the cells have equal heights in both columns and the mutual information is 0.

试一试Try this

"自信却没用"的传感器：让 TPR 和 FPR 都在 0.7 以上且相等，互信息为 0：传感器总是很有"态度"，但和门毫无关系。A confident but useless sensor: make TPR and FPR equal and both above 0.7, so the mutual information is 0: the sensor always has an "opinion" that has nothing to do with the door.
"撒谎"的传感器：让 TPR ≤ 0.1、FPR ≥ 0.9，互信息仍超过 0.5 比特：一个稳定地说反话的传感器和说真话的传感器一样有用。A lying sensor: set TPR ≤ 0.1 and FPR ≥ 0.9 and still have mutual information above 0.5 bits: a sensor that reliably says the opposite is as useful as one that reliably tells the truth.
先验也重要：把先验 $\pi$ 调到 0.95 以上，传感器最多只能提供不到 0.3 比特，因为你本来就几乎知道答案。The prior matters: set the prior $\pi$ above 0.95; the sensor can then provide under 0.3 bits at most, because you almost knew the answer already.

6再看数学The math

提示：悬停下面公式中带底色的符号，FIG 2 和 FIG 4 里对应的部分会高亮。Tip: hover the shaded symbols below and the matching parts of FIG 2 and FIG 4 light up.

熵与联合、条件熵Entropy, joint and conditional entropy

$$ H(X)=-\sum_x p(x)\log p(x)=\mathbb E\big[-\log p(X)\big],\qquad 0\le H(X)\le\log |\mathcal X|. $$ $$ H(X\mid Z)=\sum_z p(z)\,H(X\mid Z=z), $$ $$ H(X,Z)=H(Z)+H(X\mid Z)\quad\text{(chain rule)}. $$

对数以 2 为底，单位是比特；以 $e$ 为底，单位是奈特 (nat)，$1$ nat $=1.443$ 比特。机器学习里常用 nat。条件会使熵减小，平均而言知道更多总不会更糊涂：$H(X\mid Z)\le H(X)$。

With base-2 logs the unit is the bit; with base $e$ it is the nat, $1$ nat $=1.443$ bits. Machine learning mostly uses nats. Conditioning reduces entropy: on average, knowing more never makes you more confused, $H(X\mid Z)\le H(X)$.

交叉熵与 KL 散度Cross-entropy and KL divergence

$$ \hl{CE}{H(p,q)}=-\sum_x p(x)\log q(x)=\hl{H}{H(p)}+\hl{KL}{D_{\mathrm{KL}}(p\,\|\,q)}, $$ $$ D_{\mathrm{KL}}(p\,\|\,q)=\sum_x p(x)\log\frac{p(x)}{q(x)}. $$

KL 非负（吉布斯不等式）：由 $\log$ 的凹性与 Jensen 不等式，$-D_{\mathrm{KL}}=\sum_x p\log\frac qp\le\log\sum_x p\frac qp=\log\sum_{x:p>0}q\le0$，等号当且仅当 $p=q$。所以当 $q=p$ 时交叉熵最小，等于 $H(p)$。KL不对称、也不满足三角不等式，所以不是距离；若 $p(x)>0$ 而 $q(x)=0$，则 $D_{\mathrm{KL}}(p\|q)=\infty$（呼应 4.7 节"不要把概率写成 0"）。

KL is non-negative (Gibbs' inequality): by the concavity of $\log$ and Jensen, $-D_{\mathrm{KL}}=\sum_x p\log\frac qp\le\log\sum_x p\frac qp=\log\sum_{x:p>0}q\le0$, with equality iff $p=q$. Hence cross-entropy is minimized at $q=p$, where it equals $H(p)$. KL is asymmetric and violates the triangle inequality, so it is not a distance; if $p(x)>0$ while $q(x)=0$ then $D_{\mathrm{KL}}(p\|q)=\infty$ (an echo of Lesson 4.7's "never write a probability as 0").

互信息Mutual information

$$ \hl{MI}{I(X;Z)}=H(X)-H(X\mid Z)=H(Z)-H(Z\mid X) $$ $$ \phantom{I(X;Z)}=D_{\mathrm{KL}}\big(p(x,z)\,\big\|\,p(x)\,p(z)\big)\;\ge\;0. $$

它是联合分布与"假装独立"的分布之间的 KL，所以对 $X$ 和 $Z$ 对称，当且仅当两者独立时为 0。FIG 4 中"撒谎的传感器"之所以信息量最大，是因为互信息只看依赖，不看读数的"含义"。传感器的价值 = 它的读数让我们对 $X$ 的预测能力提升了多少，而不是读数字面上准不准。再往后，数据处理不等式说：对 $X\to Z\to W$，$I(X;W)\le I(X;Z)$：随便怎么加工读数，都不会凭空增加关于 $X$ 的信息。

It is the KL between the joint distribution and the "pretend they are independent" distribution, so it is symmetric in $X$ and $Z$, and zero iff the two are independent. The "lying sensor" of FIG 4 is maximally informative because mutual information looks only at dependence, not at what a reading "means". A sensor's value is how much its readings improve our ability to predict $X$, not whether they are literally correct. Beyond that, the data-processing inequality says that for $X\to Z\to W$, $I(X;W)\le I(X;Z)$: however you post-process readings, you cannot create information about $X$ out of nothing.

高斯情形：熵与 KL 的闭式The Gaussian case: closed forms of entropy and KL

$$ h\big(\mathcal N(\boldsymbol\mu,\Sigma)\big)=\tfrac12\ln\big((2\pi e)^n\det\Sigma\big), $$ $$ D_{\mathrm{KL}}\big(\mathcal N(\mu_1,\sigma_1^2)\,\|\,\mathcal N(\mu_2,\sigma_2^2)\big)=\ln\frac{\sigma_2}{\sigma_1}+\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2}-\frac12 . $$

高斯的熵只依赖 $\ln\det\Sigma$：协方差椭圆"体积"的对数。所以卡尔曼滤波每次更新，信息增益就是 $\tfrac12\ln\frac{\det P^-}{\det P}$（4.8 节：协方差椭圆变小了多少）。连续分布的这个"微分熵"可以为负，它不再是"比特数"，但差值（如 KL、互信息）仍有明确含义。

A Gaussian's entropy depends only on $\ln\det\Sigma$, the log of the covariance ellipse's "volume". So the information gain of one Kalman update is $\tfrac12\ln\frac{\det P^-}{\det P}$ (Lesson 4.8: how much the covariance ellipse shrank). This "differential entropy" of a continuous distribution can be negative and is no longer a bit count, but differences (KL, mutual information) keep a clear meaning.

7机器人中的它：下一个最佳视角In robotics: the next best view

一台机器人在未知的房间里建占据栅格地图：每个格子占据的概率是 $m_c$，一开始是 0.5（完全不知道，熵 1 比特）。它去哪个位置扫描，才能最快地弄清这张地图？

在视点 $v$ 处，传感器能读到半径 $R$ 内的每个格子（读数带噪声：真阳性率 0.9，假阳性率 0.1）。每个格子的期望信息增益就是它和读数之间的互信息 $I(m_c;z_c)$；整个视点的增益是这些格子之和（假设读数独立）：

$$ \mathrm{IG}(v)=\sum_{c\in\mathrm{view}(v)} I(m_c;z_c)=\sum_{c}\Big(H(m_c)-\mathbb E_{z_c}\big[H(m_c\mid z_c)\big]\Big). $$

已经确认的格子（概率接近 0 或 1）熵很低，再看一遍也没什么增益；最不确定的区域信息增益最大。贪心地反复"走到增益最大的视点，扫描，更新地图"，就是经典的信息增益驱动的探索。点击地图选一个视点试试；用"去最优视点"一路跟着贪心走。

A robot builds an occupancy-grid map of an unknown room: each cell's occupancy probability $m_c$ starts at 0.5 (total ignorance, entropy 1 bit). Where should it scan from to figure out the map fastest?

From a viewpoint $v$, the sensor reads every cell within radius $R$ (readings are noisy: true-positive rate 0.9, false-positive rate 0.1). The expected information gain of a cell is the mutual information $I(m_c;z_c)$ between it and its reading; a viewpoint's gain is the sum over its cells (assuming independent readings):

$$ \mathrm{IG}(v)=\sum_{c\in\mathrm{view}(v)} I(m_c;z_c)=\sum_{c}\Big(H(m_c)-\mathbb E_{z_c}\big[H(m_c\mid z_c)\big]\Big). $$

Cells already settled (probability near 0 or 1) have low entropy, so looking again gains little; the most uncertain regions have the highest gain. Greedily repeating "drive to the highest-gain viewpoint, scan, update the map" is classic information-gain-driven exploration. Click the map to pick a viewpoint; use "Go to best" to follow the greedy rule all the way.

FIG 5信息增益驱动的探索：下一个最佳视点Information-gain-driven exploration: the next best viewpoint

每个格子：青色雾越浓 = 熵越高（越不确定），实心的中性色方块 = 很可能是障碍，空白 = 很可能是空地。黄色圆点大小 = 以该格为视点的期望信息增益；绿圈是增益最大的视点，紫圈与虚线圆是你选的视点及其感知范围，蓝色小方块是机器人当前所在。点击任意格子选择视点。 Per cell: the denser the cyan haze, the higher the entropy (the more uncertain); solid neutral-coloured blocks are probably obstacles, blank is probably free. The size of an amber dot is the expected information gain of using that cell as the viewpoint; the green ring marks the best viewpoint, the purple ring and dashed circle your chosen viewpoint and its sensing range, and the small blue square is where the robot is now. Click any cell to choose a viewpoint.

试一试Try this

先点一个增益很小的视点（圆点最小的地方）去扫描，再点增益最大的：比较两次地图总熵降了多少。First scan from a viewpoint of tiny gain (smallest dot), then from the best one: compare how much the map entropy fell each time.
只用"去最优视点"：在 7 次扫描内把地图总熵降到初始的 40% 以下。Using only "Go to best", bring the total map entropy below 40% of its initial value within 7 scans.

真实系统里In real systems

这是"下一个最佳视角 (next-best-view)"与"信息增益驱动的探索"的骨架：三维重建与 NeRF 采集里的视点选择；无人机在未知环境中的前沿探索；主动 SLAM 里权衡"去探索新区域"与"回头做回环、压低位姿不确定性"；机械臂用触觉探测物体形状时选下一个触点。真实系统要考虑遮挡（射线被障碍挡住）、移动代价（增益除以路程）、读数相关性，并用分块的熵（如互信息的子模性）加速，不过核心就是 FIG 5 里这一条：选期望熵下降最多的动作。

This is the skeleton of next-best-view planning and information-gain-driven exploration: viewpoint selection when capturing data for 3D reconstruction or NeRFs; frontier exploration by drones in unknown environments; active SLAM trading off "explore new areas" against "go back for loop closures to shrink pose uncertainty"; a robot arm choosing its next touch point when probing an object's shape by touch. Real systems also account for occlusion (rays blocked by obstacles), travel cost (gain divided by path length) and correlated readings, and speed things up with submodularity, but the core is the one rule of FIG 5: pick the action with the largest expected entropy reduction.

8常见误区Common pitfalls

误区 1：把 KL 当距离Pitfall 1: treating KL as a distance

KL 不对称，也不满足三角不等式。$D_{\mathrm{KL}}(p\|q)$ 与 $D_{\mathrm{KL}}(q\|p)$ 回答的是不同的问题，拟合出来的结果大相径庭（FIG 3）。需要真正的距离或对称量时，可以用 Jensen–Shannon 散度、总变差或 Wasserstein 距离。

KL is asymmetric and violates the triangle inequality. $D_{\mathrm{KL}}(p\|q)$ and $D_{\mathrm{KL}}(q\|p)$ answer different questions and give very different fits (FIG 3). When you need a true metric or a symmetric quantity, use the Jensen–Shannon divergence, total variation or a Wasserstein distance.

误区 2：把"熵高"当成"值大"，把 0 log 0 当成错误Pitfall 2: confusing "high entropy" with "large values", and fearing 0 log 0

熵只看概率的分布，不看结果的取值。约定 $0\log0=0$（极限）。另外要注意单位：模型里的对数几乎总是自然对数 nat，而这一节为了直觉用比特，二者差一个 $\ln 2$，比较别人的数字时要先看底数。

Entropy looks only at the probabilities, not at the values of the outcomes. The convention is $0\log0=0$ (a limit). Mind the units too: code almost always uses natural logs (nats) while this lesson uses bits for intuition; they differ by a factor $\ln 2$, so check the base before comparing numbers with someone else's.

误区 3：忘记 $q$ 不能为 0Pitfall 3: forgetting that $q$ must not be 0

交叉熵里的 $\log q$ 在 $q=0$ 时是 $-\infty$。实现时要对概率做下限（$\log(q+\varepsilon)$），或者直接在 logits 上用数值稳定的 log_softmax 版交叉熵，不要先 softmax 再 log。

The $\log q$ in cross-entropy is $-\infty$ when $q=0$. In code, floor the probabilities ($\log(q+\varepsilon)$) or, better, use the numerically stable log_softmax-based cross-entropy on the logits, instead of softmax followed by log.

误区 4：信息增益最大 ≠ 最好Pitfall 4: the largest information gain is not always the best

FIG 5 的贪心规则没考虑代价：最远处的视点增益可能最大，但走过去要很久。实用的规则是"增益 ÷ 代价"或带折扣的多步规划。此外，增益是在当前模型下算的期望：如果传感器模型或先验是错的，信息增益也会误导你。

FIG 5's greedy rule ignores cost: the farthest viewpoint may have the largest gain but take ages to reach. Practical rules use "gain ÷ cost" or discounted multi-step planning. Also, the gain is an expectation under the current model; if the sensor model or the prior is wrong, the information gain will mislead you too.

误区 5：互信息大 = 传感器"准"Pitfall 5: large mutual information means an "accurate" sensor

FIG 4 的"撒谎传感器"信息量最大，因为互信息衡量的是统计依赖。你需要知道传感器的混淆矩阵（它怎样出错），才能把读数解码成信念。如果你把一个稳定说反话的传感器当成说真话的来用，后验会被推向错误的方向，即使互信息很高。

The "lying sensor" of FIG 4 is maximally informative because mutual information measures statistical dependence. You still need the sensor's confusion matrix (how it errs) to decode readings into a belief. If you use a sensor that reliably lies as if it told the truth, the posterior is pushed the wrong way even though the mutual information is high.

9小练习Check yourself

1. 三种结果的概率为 $(\tfrac12,\tfrac14,\tfrac14)$，熵是多少比特？1. Three outcomes with probabilities $(\tfrac12,\tfrac14,\tfrac14)$. What is the entropy in bits?

$H=\tfrac12\cdot1+\tfrac14\cdot2+\tfrac14\cdot2=1.5$ 比特。1.585 = $\log_2 3$ 是三者均匀时的最大值，这个分布不均匀，所以更小。也可以用"问问题"来理解：先问"是不是第一种？"（1 个问题，一半概率到此为止），否则再问一个，平均 $0.5\times1+0.5\times2=1.5$ 个问题。$H=\tfrac12\cdot1+\tfrac14\cdot2+\tfrac14\cdot2=1.5$ bits. 1.585 = $\log_2 3$ is the maximum for three uniform outcomes; this distribution is not uniform, so it is smaller. In questions: first ask "is it the first one?" (1 question, half the time you are done); otherwise ask one more; the average is $0.5\times1+0.5\times2=1.5$ questions.

2. 分类网络对一张图片的真实类别（独热标签）输出的概率为 0.25。这一个样本的交叉熵损失是多少比特？2. A classifier outputs probability 0.25 for the true class (one-hot label) of an image. What is the cross-entropy loss of this one sample, in bits?

独热标签下 $H(p)=0$，交叉熵 $=-\log_2 q_{\text{正确类}}=-\log_2 0.25=2$ 比特（用自然对数则是 $\ln4\approx1.386$ nat）。概率越接近 0，损失越爆炸：这也是为什么"自信地答错"在交叉熵下代价极高。With a one-hot label $H(p)=0$ and the cross-entropy is $-\log_2 q_{\text{correct}}=-\log_2 0.25=2$ bits ($\ln4\approx1.386$ nats in natural logs). As the probability approaches 0 the loss explodes, which is why "confidently wrong" is extremely costly under cross-entropy.

3. 关于 KL 散度，哪个说法是对的？3. Which statement about the KL divergence is correct?

非负性来自吉布斯不等式（Jensen）。它不对称（FIG 2、FIG 3 都演示了），也不满足三角不等式，所以不是距离。Non-negativity follows from Gibbs' inequality (Jensen). It is asymmetric (shown in FIG 2 and FIG 3) and violates the triangle inequality, so it is not a distance.

4. 一个传感器总是报告与真实状态相反的结果（TPR = 0，FPR = 1）。它与状态的互信息是？4. A sensor always reports the opposite of the true state (TPR = 0, FPR = 1). What is its mutual information with the state?

读到"开"就知道是"关"，反之亦然，$H(X\mid Z)=0$，所以 $I(X;Z)=H(X)-0=H(X)$。互信息从不为负，且衡量依赖而不是正确与否。（前提是你知道它在反着说。）Reading "open" means "closed" and vice versa, so $H(X\mid Z)=0$ and $I(X;Z)=H(X)-0=H(X)$. Mutual information is never negative and measures dependence, not correctness. (Provided you know it speaks backwards.)

5. 为什么"最小化训练集上的交叉熵"与"最大似然"是一回事？5. Why is "minimizing cross-entropy on the training set" the same as "maximum likelihood"?

$H(\hat p,q)=-\sum_x\hat p(x)\log q(x)=-\frac1n\sum_i\log q(x_i)$。最小化它就是最大化平均对数似然（4.6 节）。而 $H(\hat p)$ 与模型无关，所以也等价于最小化 $D_{\mathrm{KL}}(\hat p\|q)$。$H(p)$ 是否为 0 无关紧要（独热标签只是特例）。$H(\hat p,q)=-\sum_x\hat p(x)\log q(x)=-\frac1n\sum_i\log q(x_i)$. Minimizing it is maximizing the average log-likelihood (Lesson 4.6). And since $H(\hat p)$ does not depend on the model, it is also equivalent to minimizing $D_{\mathrm{KL}}(\hat p\|q)$. Whether $H(p)$ is 0 is irrelevant (one-hot labels are just a special case).

10延伸阅读Further reading

T. M. Cover & J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006, Ch. 2 — 熵、KL、互信息的标准教科书the standard textbook for entropy, KL and mutual information
D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003, Ch. 1–2, 4 — 直觉丰富，把信息论与推断、学习放在一起讲，附免费电子版rich in intuition, treating information theory together with inference and learning; free online edition
C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006, §1.6 — 信息论在机器学习中的角色：熵、KL 与互信息the role of information theory in machine learning: entropy, KL and mutual information
S. Thrun, W. Burgard & D. Fox, Probabilistic Robotics, MIT Press, 2005, Ch. 17 — 机器人探索：信息增益驱动的贪心探索robot exploration: greedy information-gain-driven exploration
3Blue1Brown, "Solving Wordle using information theory" (YouTube) — 用猜词游戏讲熵和信息增益，几乎没有公式也能讲明白entropy and information gain explained through a word game, with almost no formulas