聚类与 PCAClustering & PCA

前面几章的学习都有“标准答案”：16.1 节的回归和分类，每个输入都配着一个目标 \(y\)。可机器人每天产生的数据绝大多数没人标注。一个演示者在桌上做了 150 次抓取，日志里只有夹爪闭合时的位置，没人写下“这次抓的是杯子”；一只 15 个关节的灵巧手记录了几千个抓取姿态，没人告诉你哪些关节总是一起动；一段拾放演示是一条长长的时间序列，没人标出“这里开始抬起”。

无监督学习从数据本身找结构。本节讲两类最基本、至今仍到处在用的工具：

• 聚类：把数据分成 \(K\) 组，同组的点彼此相近。k-means 用“最近的中心”做硬划分；高斯混合模型（GMM）给每个点一个属于各组的概率，用 EM 算法训练，它建立在 4.4 节的多元高斯和 4.6 节的最大似然之上。
• 主成分分析（PCA）：找一个低维的线性子空间，让数据投影上去后损失最少。1.12 节已经用 SVD 算过点云的主方向和法向量；这里讲它的统计含义：为什么“方差最大”和“重建误差最小”是同一个问题，“解释方差”怎么读，以及该保留几个成分。

它们在机器人里随处可见：把桌面点云分割成一个个物体，把视觉特征量化成“视觉单词”用于回环检测，把演示切成阶段，把动作空间离散成几个“模式”，把高维的手部姿态压缩成两三个协同量。它们的共同弱点同样重要：k-means 只认圆团，PCA 只认平面，数据一弯就失效。这正是 18.2 节自编码器要解决的问题。

Learning in the previous chapters came with "answers": the regression and classification of Lesson 16.1 pair every input with a target \(y\). Yet most of the data a robot produces every day is never labelled. A demonstrator performs 150 grasps on a table and the log only holds where the gripper closed, not "this one was the mug"; a 15-joint dexterous hand records thousands of grasp postures and nobody says which joints always move together; a pick-and-place demonstration is a long time series and nobody marks "the lift starts here".

Unsupervised learning finds structure in the data itself. This lesson covers the two most basic families of tools, still used everywhere:

• Clustering: split the data into \(K\) groups whose members are close to each other. k-means makes a hard split by "nearest centre"; the Gaussian mixture model (GMM) gives each point a probability of belonging to each group and is trained by the EM algorithm, built on the multivariate Gaussian of Lesson 4.4 and the maximum likelihood of Lesson 4.6.
• Principal component analysis (PCA): find a low-dimensional linear subspace onto which the data projects with the least loss. Lesson 1.12 already used the SVD to get the principal directions and the normal of a point cloud; here we cover its statistical meaning: why "maximum variance" and "minimum reconstruction error" are the same problem, how to read "explained variance", and how many components to keep.

Robots use them everywhere: segmenting a tabletop point cloud into objects, quantizing visual features into "visual words" for loop closure, cutting demonstrations into phases, discretizing an action space into a few "modes", compressing high-dimensional hand postures into two or three synergies. Their shared weakness matters just as much: k-means only knows round blobs and PCA only knows flat subspaces, so both fail as soon as the data bends. That is the problem the autoencoders of Lesson 18.2 address.

在桌面上插 \(K\) 面小旗（中心 \(\hl{mu}{\boldsymbol\mu_k}\)）。然后反复做两件事：① 站队：每个数据点走到离它最近的旗子下面；② 挪旗：每面旗子挪到它队伍的平均位置（重心）。这就是 Lloyd 算法，人们口中的 k-means。它在降低一个量：每个点到自己旗子的距离平方之和

Plant \(K\) little flags on the table (centres \(\hl{mu}{\boldsymbol\mu_k}\)). Then repeat two things: ① line up: every data point walks to its nearest flag; ② move the flags: every flag moves to the average position (centroid) of its queue. That is Lloyd's algorithm, what people call k-means. It lowers one number, the sum of squared distances from each point to its own flag:

其中 \(c(n)\) 是点 \(\mathbf x_n\) 所在的组。站队这一步，在旗子不动时为每个点选了最近的旗，\(J\) 不会变大；挪旗这一步，在队伍不变时把旗子放到重心，而重心正是使队内距离平方和最小的点（第 5 节），\(J\) 也不会变大。\(J\) 只降不升，而把 \(N\) 个点分成 \(K\) 组的方式只有有限多种，所以算法一定会停。但它停下的地方只是“两步都改进不了”的地方，不一定是全局最好的分法。下图让你亲手体验这两点。

Here \(c(n)\) is the cluster of point \(\mathbf x_n\). The line-up step picks the nearest flag for every point while the flags stand still, so \(J\) cannot grow; the move step puts each flag at its queue's centroid while the queues stay fixed, and the centroid is exactly the point that minimizes the sum of squared distances within the queue (Section 5), so \(J\) cannot grow either. \(J\) only goes down, and there are only finitely many ways to split \(N\) points into \(K\) groups, so the algorithm must stop. But where it stops is only a place that neither step can improve, not necessarily the best split overall. The figure lets you feel both facts.

k-means 用欧氏距离判断“最近”，等于默认每个组都是同样大小的圆团。真实的组常常不是：同一个长条形物体上的抓取点沿物体长轴散开；一个大托盘旁边放着一个小螺丝盒。高斯混合模型换一种说法：数据是这样产生的，先以概率 \(\hl{pi}{\pi_k}\) 选一个组 \(k\)，再从这个组的高斯 \(\mathcal N(\boldsymbol\mu_k,\hl{Sigma}{\Sigma_k})\) 里抽一个点。每组有自己的位置、形状（协方差椭圆，4.4 节）和份额。

训练它的 EM 算法和 Lloyd 长得几乎一样，只是“站队”变软了：E 步算每个点属于每组的后验概率，即责任 \(\hl{gamma}{\gamma_{nk}}\)（贝叶斯公式，4.2 节）；M 步用这些责任作权重，重新估计每组的 \(\pi_k,\boldsymbol\mu_k,\Sigma_k\)（加权的最大似然，4.6 节）。每一轮 EM 都不会降低数据的对数似然（第 5 节证明）。

k-means judges "nearest" by Euclidean distance, which amounts to assuming every group is a round blob of the same size. Real groups often are not: grasps on one long object spread along its main axis; a small screw box sits next to a big tray. A Gaussian mixture model tells a different story of how the data were made: first pick a group \(k\) with probability \(\hl{pi}{\pi_k}\), then draw a point from that group's Gaussian \(\mathcal N(\boldsymbol\mu_k,\hl{Sigma}{\Sigma_k})\). Each group has its own position, shape (a covariance ellipse, Lesson 4.4) and share.

The EM algorithm that trains it looks almost like Lloyd's, except that the line-up becomes soft: the E step computes the posterior probability of each point belonging to each group, the responsibility \(\hl{gamma}{\gamma_{nk}}\) (Bayes' rule, Lesson 4.2); the M step re-estimates every group's \(\pi_k,\boldsymbol\mu_k,\Sigma_k\) with those responsibilities as weights (weighted maximum likelihood, Lesson 4.6). No EM round ever lowers the log-likelihood of the data (proved in Section 5).

FIG 1 已经看到：Lloyd 的结局取决于起点。最常见的起点有两种。随机起点（Forgy 法）：随机挑 \(K\) 个数据点当初始中心，容易把两个中心放进同一团、另一团没有中心。k-means++（Arthur 与 Vassilvitskii，2007）：第一个中心随机挑，之后每个新中心按“到已有中心的最近距离的平方” \(D(\mathbf x)^2\) 成比例地抽样，离现有中心远的点更容易被选中，于是中心自然散开。它在期望意义下保证结果不差于最优解的 \(O(\log K)\) 倍，实践中再配合多次重启取最好的一次。

另一个问题是 \(K\) 本身。\(J^\star(K)\)（\(K\) 个组时能达到的最小 \(J\)）随 \(K\) 单调下降，\(K=N\) 时为 0，所以不能拿 \(J\) 直接选 \(K\)。常用的经验法是“肘部”：画 \(J^\star(K)\) 对 \(K\)，找下降突然变缓的那个拐点。对 GMM 可以用信息准则（第 5 节的 BIC）。下图是一个货架上的 8 个格子，每格 25 次抓取。

FIG 1 showed that Lloyd's ending depends on its start. Two starts are common. Random start (Forgy's method): pick \(K\) data points at random as the initial centres; it easily puts two centres into one blob and none into another. k-means++ (Arthur and Vassilvitskii, 2007): choose the first centre at random, then draw every new centre with probability proportional to \(D(\mathbf x)^2\), the squared distance to the nearest centre chosen so far, so points far from the existing centres are favoured and the centres spread out naturally. In expectation it is guaranteed to be within an \(O(\log K)\) factor of the optimum, and in practice it is combined with several restarts, keeping the best.

The other question is \(K\) itself. \(J^\star(K)\), the smallest \(J\) reachable with \(K\) groups, decreases monotonically in \(K\) and is 0 at \(K=N\), so \(J\) alone cannot choose \(K\). A common rule of thumb is the "elbow": plot \(J^\star(K)\) against \(K\) and look for the bend where the decrease suddenly slows. For a GMM one can use an information criterion (the BIC of Section 5). The figure below is a shelf with 8 bins and 25 grasps per bin.

聚类用 \(K\) 个“原型”概括数据；降维换个角度，用更少的坐标描述每一个点。设想一台机械臂做了 120 次够取，每次记录肩关节角 \(q_1\) 和肘关节角 \(q_2\)。抬肩时人总是同时伸肘，两个角高度相关：点云是一根斜着的细长条。如果只允许用一个数描述每次够取，最好的办法是在这根长条上画一条直线 \(\hl{u}{\mathbf u}\)（单位向量），用点在这条线上的投影坐标 \(z=\mathbf u^\top(\mathbf x-\bar{\mathbf x})\) 代替原来的两个角。

线该朝哪个方向？有两种说法。一种是让投影后的坐标方差最大，保留尽可能多的“变化”；另一种是让每个点到直线的距离平方和（重建误差，即图中的残差 \(\hl{res}{\mathbf r_n}\)）最小。下图告诉你它们是同一个方向：对每个点，到均值的距离平方 = 投影长度平方 + 残差平方（勾股定理），左边与方向无关，所以投影方差最大时残差一定最小。这个方向就是协方差矩阵 \(S\) 最大特征值 \(\hl{lam}{\lambda_1}\) 的特征向量，第一主成分。

Clustering summarizes data with \(K\) "prototypes"; dimensionality reduction takes another angle and describes every point with fewer coordinates. Picture an arm making 120 reaches, each recorded as its shoulder angle \(q_1\) and elbow angle \(q_2\). Raising the shoulder always comes with extending the elbow, so the two angles are strongly correlated: the cloud is a thin slanted bar. If each reach may be described by a single number, the best idea is to draw a line \(\hl{u}{\mathbf u}\) (a unit vector) along the bar and replace the two angles by the coordinate of the projection on it, \(z=\mathbf u^\top(\mathbf x-\bar{\mathbf x})\).

Which way should the line point? There are two answers. One makes the projected coordinate's variance as large as possible, keeping as much "variation" as it can; the other makes the sum of squared distances from the points to the line (the reconstruction error, the residuals \(\hl{res}{\mathbf r_n}\) in the figure) as small as possible. The figure shows they give the same direction: for each point, squared distance to the mean = squared projection + squared residual (Pythagoras), and the left side does not depend on the direction, so the largest projected variance means the smallest residual. That direction is the eigenvector of the covariance matrix \(S\) with the largest eigenvalue \(\hl{lam}{\lambda_1}\): the first principal component.

换到“各向同性”的点云：两个方向的方差差不多（约 110 对 88），右图的曲线几乎是平的，第一主成分只解释约 56%，换一个样本（“新样本”）主轴就会大幅转动。PCA 的方向只在特征值明显分开时才有意义。

Switch to the "isotropic" cloud: the two directions have similar variances (about 110 versus 88), the curve on the right is almost flat, the first component explains only about 56%, and a new sample ("New sample") turns the axes a lot. PCA directions only mean something when the eigenvalues are clearly apart.

PCA 找的是平的子空间，k-means 找的是凸的团块（每组是一个 Voronoi 多边形）。数据一弯，两者都会出问题。下图第一组数据是夹爪沿圆碗边缘擦拭的路径（3/4 个圆）：它本质上是一维的（一个角度就能描述每个点），但一条直线只能保留约 70% 的方差。一个补救办法是分段线性：先用 k-means 把数据切成 \(K\) 段，再在每段里各做一次一维 PCA，即局部 PCA，也就是概率模型里的“混合 PCA”的简化版。第二组数据是激光雷达在一根圆柱和外圈圆墙上的点：两组是同心圆环，k-means 怎么画直线边界也分不开它们，但若换一个特征，用点到圆心的半径 \(\|\mathbf x-\mathbf c\|\)，问题立刻变成一维、可分。选对特征，线性方法就够了；核方法（核 PCA、谱聚类）和 18.2 节的自编码器，都是让机器自己学这种特征。

PCA looks for flat subspaces; k-means looks for convex blobs (each cluster is a Voronoi polygon). When the data bend, both fail. The first data set below is a gripper's path wiping around the rim of a round bowl (three quarters of a circle): it is intrinsically one-dimensional (one angle describes every point), yet a single line keeps only about 70% of the variance. One remedy is to go piecewise linear: cut the data into \(K\) pieces with k-means and run a 1D PCA inside each, a local PCA, the simplified version of what probabilistic models call a "mixture of PCAs". The second data set is lidar points on a round pillar and on the circular wall around it: two concentric rings that k-means cannot separate with any straight boundary, but switch to another feature, the radius \(\|\mathbf x-\mathbf c\|\) from the centre, and the problem becomes one-dimensional and separable. With the right features linear methods suffice; kernel methods (kernel PCA, spectral clustering) and the autoencoders of Lesson 18.2 let the machine learn such features itself.

Data \(\mathbf x_1,\dots,\mathbf x_N\in\R^D\). Write the assignment as indicators \(r_{nk}\in\{0,1\}\) with \(\sum_k r_{nk}=1\). The objective is

数据 \(\mathbf x_1,\dots,\mathbf x_N\in\R^D\)。把分组写成指示量 \(r_{nk}\in\{0,1\}\)，\(\sum_k r_{nk}=1\)。目标函数是

站队：固定 \(\boldsymbol\mu\)，\(J\) 对每个 \(n\) 分开，各自取 \(r_{nk}=1\) 给距离最小的 \(k\)，这显然是最优的。挪旗：固定 \(r\)，\(J\) 是 \(\boldsymbol\mu_k\) 的凸二次函数，令梯度为零：

Line up: with \(\boldsymbol\mu\) fixed, \(J\) separates over \(n\), and setting \(r_{nk}=1\) for the nearest \(k\) is obviously optimal. Move: with \(r\) fixed, \(J\) is a convex quadratic in \(\boldsymbol\mu_k\); set its gradient to zero:

即重心。另一种看法：对任意点 \(\mathbf m\)，\(\sum_{n\in k}\|\mathbf x_n-\mathbf m\|^2=\sum_{n\in k}\|\mathbf x_n-\boldsymbol\mu_k\|^2+N_k\|\boldsymbol\mu_k-\mathbf m\|^2\)，所以偏离重心只会多出一项。两步都精确最优，\(J\) 单调不增；分组方式至多 \(K^N\) 种，且一次真正的改变会严格降低 \(J\)，所以有限步内收敛。若某组变空（FIG 1 的“角落起点”），重心无定义，本页让它留在原地；实际库里常把它重新放到离现有中心最远的点上。

找全局最优的 k-means 划分在一般情况下是 NP 难的，所以只能靠好的起点和重启。k-means++ 的抽样规则是：已选中心集合为 \(\mathcal C\) 时，下一个中心取 \(\mathbf x_n\) 的概率为

the centroid. Another way to see it: for any point \(\mathbf m\), \(\sum_{n\in k}\|\mathbf x_n-\mathbf m\|^2=\sum_{n\in k}\|\mathbf x_n-\boldsymbol\mu_k\|^2+N_k\|\boldsymbol\mu_k-\mathbf m\|^2\), so leaving the centroid only adds a term. Both steps are exactly optimal and \(J\) never increases; there are at most \(K^N\) assignments and a genuine change strictly lowers \(J\), so it converges in finitely many steps. If a cluster becomes empty (FIG 1's "Corner start") its centroid is undefined; this page leaves it where it is, while real libraries often move it to the point farthest from the current centres.

Finding the globally optimal k-means partition is NP-hard in general, so good starts and restarts are all one has. The k-means++ rule: with the set of chosen centres \(\mathcal C\), the next centre is \(\mathbf x_n\) with probability

其中 \(J_0\) 是抽样得到的初始中心本身的代价：右边的界对它就已经成立（Arthur 与 Vassilvitskii，2007），之后的 Lloyd 迭代只会让 \(J\) 更小。

Here \(J_0\) is the cost of the sampled seeds themselves: the bound on the right already holds for them (Arthur and Vassilvitskii, 2007), and the Lloyd iterations that follow can only lower \(J\).

对数里面有求和，导数为零的方程没有闭式解。EM 的做法是引入隐变量“这个点来自哪一组”。E 步：用当前参数算后验（责任）

The sum inside the logarithm means the zero-gradient equations have no closed-form solution. EM introduces the hidden variable "which group did this point come from". E step: compute the posterior (the responsibilities) with the current parameters,

M 步：把责任当作“每个点有 \(\gamma_{nk}\) 份属于第 \(k\) 组”，做加权的最大似然，记 \(N_k=\sum_n\gamma_{nk}\)：

M step: treat point \(n\) as belonging to group \(k\) with weight \(\gamma_{nk}\) and do weighted maximum likelihood; with \(N_k=\sum_n\gamma_{nk}\):

为什么似然不降：对任意满足 \(\sum_k q_{nk}=1\) 的权重 \(q_{nk}>0\)，由于对数是凹函数（Jensen 不等式），

Why the likelihood never drops: for any weights \(q_{nk}>0\) with \(\sum_k q_{nk}=1\), concavity of the logarithm (Jensen's inequality) gives

当 \(q_{nk}=\gamma_{nk}\) 时等号成立（E 步让下界在当前参数处贴住似然），M 步再对参数最大化这个下界（它对每组是可以闭式求解的加权高斯似然）。似然 ≥ 新下界 ≥ 旧下界 = 旧似然，所以每一轮都不降。这个下界在 19.1 节的 VAE 里会再次出现，那里叫 ELBO。与 k-means 的关系：令所有 \(\Sigma_k=\sigma^2I\)、\(\pi_k=1/K\) 固定，让 \(\sigma\to0\)，责任变成 0 或 1（离最近中心的那组得 1），M 步只剩更新均值：EM 退化为 k-means。

两个坑。一是奇异性：如果某个高斯的均值正好落在一个数据点上、方差趋于 0，该点的密度趋于无穷，似然无上界。FIG 2 给每个协方差加了 \(\varepsilon I\)，\(\varepsilon=0.001\) cm²（这让 M 步不再是严格的极大化，但在本页数据上对数似然的下降不超过 \(10^{-7}\)）；更严格的做法是给协方差加先验做 MAP（4.6 节）。二是选 K：似然随 K 单调上升。贝叶斯信息准则 \(\mathrm{BIC}=-2\log p_{\hat\theta}(\mathcal D)+P\ln N\)（\(P\) 为自由参数个数：2D 中每组有 2 个均值和 3 个全协方差参数，共 5 个，再加 \(K-1\) 个份额）取最小，就在拟合与复杂度之间做了折中。

with equality when \(q_{nk}=\gamma_{nk}\) (the E step makes the bound touch the likelihood at the current parameters), and the M step maximizes that bound over the parameters (for each group it is a weighted Gaussian likelihood with a closed-form maximizer). Likelihood ≥ new bound ≥ old bound = old likelihood, so no round goes down. This bound returns in the VAE of Lesson 19.1, where it is called the ELBO. The link to k-means: fix all \(\Sigma_k=\sigma^2I\) and \(\pi_k=1/K\) and let \(\sigma\to0\); the responsibilities become 0 or 1 (1 for the nearest centre), the M step only updates the means, and EM turns into k-means.

Two traps. The first is singularity: if a Gaussian's mean sits exactly on one data point and its variance goes to 0, that point's density goes to infinity and the likelihood is unbounded. FIG 2 adds \(\varepsilon I\) with \(\varepsilon=0.001\) cm² to every covariance (so the M step is no longer an exact maximization, but on this page's data the log-likelihood never drops by more than \(10^{-7}\)); the cleaner fix is a prior on the covariances, i.e. MAP (Lesson 4.6). The second is choosing K: the likelihood rises monotonically with K. Minimizing the Bayesian information criterion \(\mathrm{BIC}=-2\log p_{\hat\theta}(\mathcal D)+P\ln N\) (\(P\) free parameters: in 2D, 5 per group, 2 for the mean and 3 for a full covariance, plus \(K-1\) shares) trades fit against complexity.

记均值 \(\bar{\mathbf x}=\frac1N\sum_n\mathbf x_n\)，把中心化后的数据按行堆成 \(N\times D\) 矩阵 \(\tilde X\)，样本协方差 \(S=\frac1N\tilde X^\top\tilde X\)（统计软件常用 \(\frac1{N-1}\)，只差一个比例，方向不变）。沿单位向量 \(\mathbf u\) 的投影方差为 \(\mathbf u^\top S\mathbf u\)。在 \(\|\mathbf u\|=1\) 的约束下最大化它（拉格朗日乘子，6.7 节）：

With the mean \(\bar{\mathbf x}=\frac1N\sum_n\mathbf x_n\), stack the centred data as rows of an \(N\times D\) matrix \(\tilde X\); the sample covariance is \(S=\frac1N\tilde X^\top\tilde X\) (statistics packages often use \(\frac1{N-1}\), a mere scale factor that leaves the directions unchanged). The variance of the projection onto a unit vector \(\mathbf u\) is \(\mathbf u^\top S\mathbf u\). Maximize it subject to \(\|\mathbf u\|=1\) (Lagrange multipliers, Lesson 6.7):

所以驻点都是特征向量，投影方差就是特征值，最大值在 \(\hl{lam}{\lambda_1}\) 处（1.11 节：对称矩阵的特征向量正交，特征值为实数且这里非负）。误差的那一面：点 \(\tilde{\mathbf x}_n\) 在直线上的重建是 \(\mathbf u\mathbf u^\top\tilde{\mathbf x}_n\)，

So the stationary points are the eigenvectors, the projected variance is the eigenvalue, and the maximum is at \(\hl{lam}{\lambda_1}\) (Lesson 1.11: a symmetric matrix has orthogonal eigenvectors and real eigenvalues, here non-negative). The error side: the reconstruction of \(\tilde{\mathbf x}_n\) on the line is \(\mathbf u\mathbf u^\top\tilde{\mathbf x}_n\), and

这就是 FIG 4 的“勾股定理”：残差均方 = 总方差 − 投影方差。推广到 \(q\) 个成分：取 \(W_q=[\mathbf v_1,\dots,\mathbf v_q]\)（前 \(q\) 个特征向量），编码 \(\mathbf z=W_q^\top(\mathbf x-\bar{\mathbf x})\)，解码 \(\hat{\mathbf x}=\bar{\mathbf x}+W_q\mathbf z\)，均方重建误差恰为被丢掉的特征值之和，“解释方差比” \(\eta_q\) 为保留的那部分：

This is FIG 4's "Pythagoras": mean squared residual = total variance − projected variance. For \(q\) components take \(W_q=[\mathbf v_1,\dots,\mathbf v_q]\) (the first \(q\) eigenvectors), encode \(\mathbf z=W_q^\top(\mathbf x-\bar{\mathbf x})\) and decode \(\hat{\mathbf x}=\bar{\mathbf x}+W_q\mathbf z\); the mean squared reconstruction error is exactly the sum of the discarded eigenvalues, and the "explained variance ratio" \(\eta_q\) is the part kept:

与 SVD 的关系（1.12 节）：\(\tilde X=U\Sigma V^\top\)，则 \(S=\frac1N V\Sigma^2V^\top\)，所以主方向就是 \(V\) 的列，\(\lambda_i=\sigma_i^2/N\)；1.12 节的“\(\sigma_i/\sqrt N\) 是沿该方向的标准差”说的正是 \(\sqrt{\lambda_i}\)。数值上直接对 \(\tilde X\) 做 SVD 比先算 \(S\) 再求特征值更稳（不会把条件数平方），\(N\ll D\) 时（例如图像）也更省。最小重建误差这件事就是 1.12 节的 Eckart–Young 定理：秩 \(q\) 截断 SVD 是最好的秩 \(q\) 近似。

标准化：PCA 依赖单位。若特征的单位不同（弧度与牛顿），应先把每一维除以自己的标准差，相当于对相关矩阵做 PCA。白化：用 \(\Lambda_q^{-1/2}W_q^\top(\mathbf x-\bar{\mathbf x})\) 得到协方差为单位阵的坐标。概率 PCA（Tipping 与 Bishop，1999）把 PCA 写成一个线性高斯隐变量模型 \(\mathbf x=W\mathbf z+\bar{\mathbf x}+\boldsymbol\epsilon\)，\(\mathbf z\sim\mathcal N(0,I)\)，\(\boldsymbol\epsilon\sim\mathcal N(0,\sigma^2I)\)，其最大似然解张成同一个主子空间。把线性的编码、解码换成神经网络，就是 18.2 节的自编码器；再把隐变量当成概率分布，就是 19.1 节的 VAE。

Relation to the SVD (Lesson 1.12): with \(\tilde X=U\Sigma V^\top\), \(S=\frac1N V\Sigma^2V^\top\), so the principal directions are the columns of \(V\) and \(\lambda_i=\sigma_i^2/N\); Lesson 1.12's "\(\sigma_i/\sqrt N\) is the standard deviation along that direction" is exactly \(\sqrt{\lambda_i}\). Numerically, an SVD of \(\tilde X\) is more stable than forming \(S\) and taking its eigenvalues (it does not square the condition number), and cheaper when \(N\ll D\) (images, say). The minimum-reconstruction-error property is Lesson 1.12's Eckart–Young theorem: the rank-\(q\) truncated SVD is the best rank-\(q\) approximation.

Standardization: PCA depends on units. If features have different units (radians and newtons), first divide each dimension by its own standard deviation, which amounts to PCA of the correlation matrix. Whitening: \(\Lambda_q^{-1/2}W_q^\top(\mathbf x-\bar{\mathbf x})\) gives coordinates with identity covariance. Probabilistic PCA (Tipping and Bishop, 1999) writes PCA as a linear-Gaussian latent-variable model \(\mathbf x=W\mathbf z+\bar{\mathbf x}+\boldsymbol\epsilon\), \(\mathbf z\sim\mathcal N(0,I)\), \(\boldsymbol\epsilon\sim\mathcal N(0,\sigma^2I)\), whose maximum-likelihood solution spans the same principal subspace. Replace the linear encoder and decoder by neural networks and you get the autoencoders of Lesson 18.2; treat the latent as a probability distribution and you get the VAE of Lesson 19.1.

9.3 节提到过“协同”：人手的抓取姿态高度相关，少数几个主成分就能解释大部分方差（Santello、Flanders 与 Soechting，1998）。Ciocarlie 与 Allen（2009）把这件事用到机器人上：在几个主成分张成的“特征抓取”（eigengrasp）空间里搜索抓取，而不是在全部关节空间里搜索，让多指灵巧手的抓取规划快得多。下图用一只示意的 15 关节手（5 根手指，每根 3 个屈曲角）演示这个流程。数据是模拟的：80 个抓取姿态由三种“协同”（整手握拳、拇指与食指捏合、尺侧手指与桡侧手指的反向运动）随机组合再加 3° 的独立噪声生成，PCA 并不知道这一点，它要从 80 × 15 的数据里自己找出来。

Lesson 9.3 mentioned "synergies": human grasp postures are strongly correlated, and a few principal components explain most of their variance (Santello, Flanders and Soechting, 1998). Ciocarlie and Allen (2009) carried this over to robots: search for grasps in the "eigengrasp" space spanned by a few principal components instead of the full joint space, which makes grasp planning for multi-fingered hands much faster. The figure walks through this on a schematic 15-joint hand (5 fingers, 3 flexion angles each). The data are simulated: 80 grasp postures generated as random combinations of three "synergies" (whole-hand closing, a thumb–index pinch, and the ulnar fingers moving against the radial ones) plus 3° of independent noise. PCA is not told any of this; it has to find it in the 80 × 15 data on its own.

第一个成分解释约 74%，前两个约 95%，前三个约 98.4%，之后每个成分只剩约 0.2%：碎石图（scree plot）在第 3 个成分之后变平，这就是“该保留几个”的读法。注意 PCA 找到的方向只是张成同一个子空间，未必和生成数据的三种协同一一对应：主成分彼此正交，而我们设计的三种协同并不正交，所以 \(\mathbf v_1\) 是“握拳”混了一点其他成分（例如拇指几乎不参与）。

The first component explains about 74%, the first two about 95%, the first three about 98.4%, and each later one only about 0.2%: the scree plot goes flat after the third component, which is how one reads "how many to keep". Note that the directions PCA finds only span the same subspace and need not match the three generating synergies one to one: principal components are mutually orthogonal while the synergies we designed are not, so \(\mathbf v_1\) is "closing" mixed with a little of the others (the thumb, for one, barely takes part).