对比学习：SimCLR 与 CLIPContrastive Learning: SimCLR & CLIP

一台装配机器人的腕部相机每秒拍 30 张图，一天就是几百万张。几乎没有一张带标签：没人告诉它“这是 L 形支架”“这时夹爪离目标还有 3 cm”。可下游的一切（抓取、模仿学习、强化学习的奖励）都需要一个好的表示：把图像变成一个向量，向量只记住任务关心的东西（零件是什么、夹爪在哪），而对光照、背景、相机视角这些干扰因素不敏感。

18.2 节的自编码器靠重建每个像素来学表示，于是它必须把光照、阴影、背景纹理也一并记下：像素里的大部分信息恰恰是干扰。对比学习换了一个问题：不重建，而是比较。给网络一张图的两个变体（换了光照、加了噪声），再混进一堆别的图，让它认出哪两个来自同一个场景。为了认得准，网络必须抓住在变体之间不变、而在不同场景之间不同的东西。哪些东西算“不变”，由你选择的变换决定：这是对比学习最核心、也最容易被忽视的一点。

把“同一场景的另一个视角”换成“描述这张图的一句话”，就得到 CLIP：图像和文字被映射进同一个空间，机器人可以直接拿“蓝色的 L 形支架”去和相机画面比相似度，连训练时没见过的类别也能认（零样本分类）。把“另一个视角”换成“同一段视频里相邻的时刻”或“另一台相机同一时刻的画面”，就得到 TCN、R3M、VIP 这类机器人视觉表示，甚至可以直接用表示空间里离目标的距离当奖励。

The wrist camera of an assembly robot takes 30 images a second, millions a day. Almost none carry a label: nobody tells it "this is an L-bracket" or "the gripper is 3 cm from the goal". Yet everything downstream (grasping, imitation learning, rewards for reinforcement learning) needs a good representation: a map from an image to a vector that keeps what the task cares about (which part, where the gripper is) and is insensitive to nuisance factors such as lighting, background and camera viewpoint.

The autoencoders of Lesson 18.2 learn by reconstructing every pixel, so they must also memorize the lighting, the shadows and the background texture: most of the information in pixels is exactly the nuisance. Contrastive learning asks a different question: not reconstruct, but compare. Give the network two variants of one image (relit, with added noise), mix in a crowd of other images, and ask it to recognize which two come from the same scene. To answer well, the network must capture what stays the same across the variants and differs between scenes. Which things count as "the same" is decided by the transformations you choose: the most central and the most overlooked point of contrastive learning.

Replace "another view of the same scene" by "a sentence describing the image" and you get CLIP: images and text are mapped into one space, and a robot can compare "blue L-bracket" directly with its camera image, even for categories it never saw during training (zero-shot classification). Replace "another view" by "a nearby moment in the same video" or "another camera at the same instant" and you get robot visual representations such as TCN, R3M and VIP, where even the distance to a goal in representation space can serve as a reward.

编码器 \(f_\theta\) 把每张图变成一个向量，再归一化到单位球面上：\(\mathbf z=f_\theta(x)/\|f_\theta(x)\|\)。两张图的相似度就是 1.7 节的余弦相似度 \(s=\mathbf z^\top\mathbf z'\in[-1,1]\)。现在拿一个锚点 \(\mathbf z\)，它的正样本 \(\hl{pos}{\mathbf z^+}\)（同一场景的另一个变体），以及若干负样本 \(\hl{neg}{\mathbf z^-_j}\)（别的场景）。把每个候选的相似度除以温度 \(\hl{tau}{\tau}\) 当作 logit，做一次 softmax：这就是一个“\(N+1\) 选 1”的分类器，正确答案是正样本。它的交叉熵（16.1 节）就是 InfoNCE 损失：

The encoder \(f_\theta\) turns each image into a vector and normalizes it onto the unit sphere: \(\mathbf z=f_\theta(x)/\|f_\theta(x)\|\). The similarity of two images is the cosine similarity of Lesson 1.7, \(s=\mathbf z^\top\mathbf z'\in[-1,1]\). Now take an anchor \(\mathbf z\), its positive \(\hl{pos}{\mathbf z^+}\) (another variant of the same scene) and a few negatives \(\hl{neg}{\mathbf z^-_j}\) (other scenes). Divide each candidate's similarity by a temperature \(\hl{tau}{\tau}\), treat it as a logit and take a softmax: this is an "\(N+1\)-way" classifier whose right answer is the positive. Its cross-entropy (Lesson 16.1) is the InfoNCE loss:

（本节用 \(\mathbf z\) 表示嵌入向量，logit 写成 \(\ell=s/\tau\)，与 16.1 节把 logit 记作 \(z\) 的写法不同。）损失要变小，只有两条路：让锚点更像正样本（拉近），或者让它更不像负样本（推远）。下面直接拖动单位圆上的点，看分类器给正样本多少概率，以及锚点受到的拉力与推力。

(In this lesson \(\mathbf z\) is an embedding vector and a logit is written \(\ell=s/\tau\), unlike Lesson 16.1 where logits were called \(z\).) The loss can only shrink in two ways: make the anchor more like the positive (pull), or less like the negatives (push). Drag the points on the unit circle below and watch how much probability the classifier gives the positive, and which pulls and pushes act on the anchor.

SimCLR（Chen 等，2020）把上面的分类器变成一个完整的训练配方。每一步取 \(N\) 张图，每张图随机增广两次，得到 \(2N\) 个视图；同一张图的两个视图互为正样本，批次里其余 \(2N-2\) 个视图全部当负样本，不需要任何标签。我们的“数据集”是一台俯视托盘的相机拍的 8×8 灰度图：托盘上放着四种零件之一，方块、横销、竖销和 L 形支架，位置有 ±1 像素的抖动，每张图的环境光、零件亮度和光照梯度（光从哪边来）都不同。网络从来看不到零件类别；类别只在评估时用来给点上色和打分。

先看增广。真实 SimCLR 最重要的两种增广是随机裁剪和颜色失真；我们的编码器是一个小 MLP，学不会平移不变性（那是 17.1 节卷积网络的强项），所以这里用三种适合它的增广：光照抖动（随机改变增益、平移环境光，并叠加一个方向随机的光照梯度）、传感器噪声、随机遮挡（抹掉一个 2×2 的块）。

SimCLR (Chen et al., 2020) turns the classifier above into a complete training recipe. Each step takes \(N\) images and augments each one twice at random, giving \(2N\) views; the two views of the same image are each other's positive, and the other \(2N-2\) views in the batch are all negatives. No labels are needed. Our "dataset" comes from a camera looking down at a tray: 8×8 grayscale images showing one of four parts, a block, a horizontal peg, a vertical peg or an L-bracket, jittered by ±1 pixel, each image with its own ambient light, part brightness and lighting gradient (which side the light comes from). The network never sees the part class; the class is only used at evaluation time, to colour the points and score them.

First the augmentations. The two most important ones in the real SimCLR are random cropping and colour distortion; our encoder is a small MLP that cannot learn translation invariance (the strength of the convolutional networks of Lesson 17.1), so we use three augmentations that suit it: lighting jitter (randomly changes the gain, shifts the ambient light and adds a lighting gradient in a random direction), sensor noise, and random erasing (wipes out a 2×2 patch).

这正是要点：增广让预训练任务在像素空间里变难，网络只能去找光照改变不了的东西，也就是零件的形状。下面真正开始训练。编码器是 64 → 32 → 32 → 3 的 tanh MLP，输出归一化后落在三维单位球面上（可以拖动旋转）；损失是对 \(2N\) 个视图求平均的 InfoNCE（SimCLR 称之为 NT-Xent）。每 20 步，在 200 张固定的测试图（不增广）上评估一次：用嵌入做 5-近邻分类，看能否猜对零件类别，以及能否猜对光照方向（四个象限之一）。前者高、后者接近瞎猜的 25%，说明表示学到了零件而忽略了光照。

That is the point: augmentations make the pretext task hard in pixel space, so the network has to find what lighting cannot change, the shape of the part. Now train for real. The encoder is a 64 → 32 → 32 → 3 tanh MLP whose normalized output lies on the 3D unit sphere (drag to rotate it); the loss is InfoNCE averaged over the \(2N\) views (SimCLR calls it NT-Xent). Every 20 steps it is evaluated on 200 fixed test images (not augmented): a 5-nearest-neighbour classifier on the embeddings tries to guess the part class, and separately the lighting direction (one of four quadrants). A high first number and a second near the blind-guess 25% mean that the representation learned the part and ignored the lighting.

为什么“认出搭档”能学到有用的东西？4.11 节把互信息 \(I(X;Y)\) 定义为“知道 \(Y\) 之后，关于 \(X\) 的不确定性减少了多少”。van den Oord 等（2018）证明：用 \(N\) 个候选（1 个正样本、\(N-1\) 个负样本）训练出的 InfoNCE 损失给出互信息的一个下界 \(I(X;Y)\ge\hl{logn}{\log N}-L\)。最小化损失就是在最大化两个视图之间互信息的下界。可这个下界有个硬伤：\(L\ge0\)，所以估计值永远不超过 \(\log N\)。下图用两组相关的四维高斯变量 \(X,Y\)（真实互信息可以精确算出）和最好的打分函数（真实的密度比）来检验：即使打分函数完美，估计也撞上 \(\log N\) 的天花板。

Why does "spot your partner" learn something useful? Lesson 4.11 defined the mutual information \(I(X;Y)\) as how much knowing \(Y\) reduces the uncertainty about \(X\). Van den Oord et al. (2018) showed that the InfoNCE loss with \(N\) candidates (1 positive, \(N-1\) negatives) yields a lower bound \(I(X;Y)\ge\hl{logn}{\log N}-L\). Minimizing the loss maximizes a lower bound on the mutual information between the two views. But the bound has a hard limit: \(L\ge0\), so the estimate never exceeds \(\log N\). The figure checks this with two correlated 4-dimensional Gaussian variables \(X,Y\) (whose mutual information is known exactly) and the best possible critic (the true density ratio): even with a perfect critic, the estimate hits the \(\log N\) ceiling.

CLIP（Radford 等，2021）把正样本对从“同一张图的两个增广”换成“一张图和描述它的文字”。它有两个编码器：图像编码器 \(f\) 和文本编码器 \(g\)，都输出单位向量。一个批次里有 \(N\) 对图文，算出 \(N\times N\) 的相似度矩阵，对角线是正确配对。损失是两个方向的 InfoNCE 之和：每张图在 \(N\) 句话里找自己的描述（矩阵的行），每句话在 \(N\) 张图里找自己的图（矩阵的列）。温度也是一个可学习的参数。训练完，零样本分类不需要再训练任何分类器：把每个类别写成一句话（“一个蓝色的 L 形支架”），用文本编码器编码，选和图像余弦相似度最高的那句。

下面是一个缩小版：彩色 8×8 图像，红、绿、蓝三种颜色 × 四种零件 = 12 种组合；“文本”是两个词（颜色词 + 零件词）的词袋，文本编码器是 7 → 16 → 8 的 MLP，图像编码器是 192 → 32 → 8 的 MLP。关键的考验：“蓝色 L 形支架”和“红色竖销”这两种组合在训练中一次都不出现，模型只见过蓝色的别的零件、别的颜色的 L 形支架。和真实相机一样，三种颜色在 RGB 通道上有重叠（红色是 (1, 0.4, 0.4)，依此类推）。

CLIP (Radford et al., 2021) replaces the positive pair "two augmentations of one image" with "an image and the text describing it". It has two encoders, an image encoder \(f\) and a text encoder \(g\), both producing unit vectors. A batch of \(N\) image–text pairs gives an \(N\times N\) similarity matrix whose diagonal holds the correct pairings. The loss is the sum of InfoNCE in both directions: each image looks for its caption among the \(N\) sentences (the rows of the matrix) and each sentence looks for its image among the \(N\) images (the columns). The temperature is a learned parameter too. After training, zero-shot classification needs no classifier training at all: write each class as a sentence ("a blue L-bracket"), encode it with the text encoder, and pick the sentence with the highest cosine similarity to the image.

Below is a miniature version: colour 8×8 images, three colours red, green, blue × four parts = 12 combinations; the "text" is a bag of two words (a colour word + a part word), the text encoder a 7 → 16 → 8 MLP and the image encoder a 192 → 32 → 8 MLP. The key test: the combinations "blue L-bracket" and "red vertical peg" never appear in training; the model has only seen other blue parts and L-brackets of other colours. As with a real camera, the three colours overlap in the RGB channels (red is (1, 0.4, 0.4), and so on).

记候选集合为 \(\{\mathbf z_0=\mathbf z^+,\mathbf z_1,\dots,\mathbf z_N\}\)，logit \(\ell_j=\mathbf z^\top\mathbf z_j/\tau\)，softmax 概率 \(p_j=e^{\ell_j}/\sum_k e^{\ell_k}\)。损失 \(L=-\ell_0+\log\sum_k e^{\ell_k}\)。对锚点求导（16.1 节 softmax 交叉熵的“\(p-y\)”规则再用一次）：

Write the candidates as \(\{\mathbf z_0=\mathbf z^+,\mathbf z_1,\dots,\mathbf z_N\}\), the logits as \(\ell_j=\mathbf z^\top\mathbf z_j/\tau\) and the softmax probabilities as \(p_j=e^{\ell_j}/\sum_k e^{\ell_k}\). The loss is \(L=-\ell_0+\log\sum_k e^{\ell_k}\). Differentiating with respect to the anchor (the "\(p-y\)" rule of softmax cross-entropy from Lesson 16.1, once more):

第一项是对正样本的拉力，求和里的每一项是对一个负样本的推力（悬停公式，FIG 1 的箭头会加粗）。第二个式子是通过归一化 \(\mathbf z=\mathbf u/\|\mathbf u\|\) 反传：只有与球面相切的分量留下来，这就是 FIG 1 里琥珀色合力沿切向的原因（代码在页面里用 16.3 节的反向传播接着往网络里传）。正样本的 \(\partial L/\partial\mathbf z^+\) 和每个负样本的 \(\partial L/\partial\mathbf z_j^-\) 同理，所以负样本也被推离锚点。

温度的两个极限。\(\tau\to0\)：softmax 变成 max，\(L\approx\max\big(0,\max_j s_j^- -s^+\big)/\tau\)，只惩罚比正样本更像锚点的那个最难负样本，就是带零间隔的三元组损失。\(\tau\to\infty\)：一阶展开 \(\log\sum_k e^{\ell_k}\approx\log(N+1)+\bar\ell\)，于是 \(L\approx\log(N+1)-(s^+-\bar s)/\tau\)，\(\bar s\) 是全部 \(N+1\) 个相似度的平均：所有负样本被同等地推开，难样本不再特殊。实践中 \(\tau\) 通常在 0.05 到 0.5 之间，CLIP 则把它当参数从 0.07 开始学。

The first term is the pull towards the positive and each term of the sum a push away from one negative (hover the formula and the arrows in FIG 1 thicken). The second formula backpropagates through the normalization \(\mathbf z=\mathbf u/\|\mathbf u\|\): only the component tangent to the sphere survives, which is why the amber resultant in FIG 1 runs along the tangent (the page's code then continues into the network with the backpropagation of Lesson 16.3). The positive's \(\partial L/\partial\mathbf z^+\) and each negative's \(\partial L/\partial\mathbf z_j^-\) follow the same way, so the negatives are also pushed away from the anchor.

The two limits of the temperature. \(\tau\to0\): the softmax becomes a max and \(L\approx\max\big(0,\max_j s_j^- -s^+\big)/\tau\), which only penalizes the hardest negative, the one more similar to the anchor than the positive: a triplet loss with zero margin. \(\tau\to\infty\): to first order \(\log\sum_k e^{\ell_k}\approx\log(N+1)+\bar\ell\), so \(L\approx\log(N+1)-(s^+-\bar s)/\tau\) with \(\bar s\) the mean of all \(N+1\) similarities: every negative is pushed alike and hard ones are no longer special. In practice \(\tau\) is usually between 0.05 and 0.5, and CLIP learns it as a parameter starting from 0.07.

Wang 与 Isola（2020）证明，当负样本数 \(N\to\infty\) 时，InfoNCE 分成两项：

Wang and Isola (2020) proved that as the number of negatives \(N\to\infty\), InfoNCE splits into two terms:

对齐项希望正样本对重合；均匀项是一个“排斥势”，在球面上的均匀分布处最小（把鼠标停在这两个词上，FIG 3 会画出几对正样本视图的连线，或所有测试嵌入的平均向量）。只留对齐项，最优解就是常数函数：每张图都输出同一个 \(\mathbf z\)，对齐损失为 0，这正是 FIG 3 “只拉近”时看到的塌缩，\(R\to1\)。均匀项在塌缩处最大（所有 \(\mathbf z^\top\mathbf z^-=1\)），所以负样本把塌缩从“最优”变成了“最差”。对有限批次，塌缩处的损失恰好等于瞎猜：SimCLR 的 \(2N\) 个视图里每个锚点有 \(2N-1\) 个候选，\(L=\log(2N-1)\)。

不用负样本也能防塌缩，但要换别的机制：BYOL 和 SimSiam 用预测头加停止梯度（BYOL 还用动量目标编码器），DINO 用对教师输出的中心化与锐化，VICReg 直接惩罚各维方差过小。这些“非对比”方法留给 18.4 节。

The alignment term wants positive pairs to coincide; the uniformity term is a "repulsive potential" minimized by the uniform distribution on the sphere (hover over these two words and FIG 3 draws lines between a few positive pairs of views, or the mean vector of all test embeddings). Keep only the alignment term and the optimum is a constant function: every image outputs the same \(\mathbf z\) and the alignment loss is 0. That is the collapse seen with "pull only" in FIG 3, \(R\to1\). The uniformity term is largest at collapse (every \(\mathbf z^\top\mathbf z^-=1\)), so the negatives turn collapse from "optimal" into "worst". For a finite batch the loss at collapse equals the blind guess exactly: among SimCLR's \(2N\) views each anchor has \(2N-1\) candidates, so \(L=\log(2N-1)\).

Collapse can also be prevented without negatives, through other mechanisms: BYOL and SimSiam use a predictor head plus a stop-gradient (BYOL also a momentum target encoder), DINO centres and sharpens the teacher's outputs, and VICReg directly penalizes dimensions whose variance is too small. These "non-contrastive" methods are left to Lesson 18.4.

取 \(N\) 对样本 \((x_i,y_i)\sim p(x,y)\)，对任意打分函数 \(c(x,y)\)，下式成立（van den Oord 等 2018 提出，Poole 等 2019 给出严格证明）：

Take \(N\) pairs \((x_i,y_i)\sim p(x,y)\). For any critic \(c(x,y)\) the following holds (proposed by van den Oord et al. 2018, proved rigorously by Poole et al. 2019):

直觉：给定 \(x\)，在 \(N\) 个候选 \(y\) 里猜哪个是它的搭档，贝叶斯最优的猜法就是按密度比 \(p(y\mid x)/p(y)\) 打分，这正是 \(c^\star\)；只依赖 \(x\) 的项 \(b(x)\) 在 softmax 里抵消。FIG 4 用的就是这个 \(c^\star\)：对相关高斯，\(\log p(y\mid x)-\log p(y)\) 是 \(\frac{\rho}{1-\rho^2}\mathbf x^\top\mathbf y\) 加上只依赖 \(\mathbf y\) 和只依赖 \(\mathbf x\) 的二次项。由于 \(L_{\mathrm{NCE}}\ge0\)，右边不可能超过 \(\log N\)：要“看见” \(k\) nats 的互信息，至少需要 \(N\approx e^k\) 个候选。

Intuition: given \(x\), guess which of \(N\) candidates \(y\) is its partner; the Bayes-optimal guess scores each candidate by the density ratio \(p(y\mid x)/p(y)\), which is \(c^\star\), and any term \(b(x)\) that depends on \(x\) alone cancels in the softmax. FIG 4 uses exactly this \(c^\star\): for correlated Gaussians, \(\log p(y\mid x)-\log p(y)\) is \(\frac{\rho}{1-\rho^2}\mathbf x^\top\mathbf y\) plus quadratic terms in \(\mathbf y\) alone and in \(\mathbf x\) alone. Since \(L_{\mathrm{NCE}}\ge0\), the right side can never exceed \(\log N\): to "see" \(k\) nats of mutual information you need at least \(N\approx e^k\) candidates.

SimCLR 的编码器 \(f_\theta\)（论文里是 ResNet）后面接一个小的 MLP 投影头 \(g\)，损失作用在 \(\mathbf z\) 上，下游任务却用投影之前的 \(\mathbf h\)：投影头会主动丢掉与增广有关的信息（比如颜色），而 \(\mathbf h\) 还保留着，所以 \(\mathbf h\) 上的线性分类效果更好。评价表示的标准做法是线性探测：冻结编码器，只在 \(\mathbf h\) 上训练一个线性分类器（21.1 节会把它和全量微调比较）。我们的小 MLP 没有单独的投影头，直接在 \(\mathbf z\) 上做近邻评估。CLIP 的式子里 \(\mathbf u_i\) 是第 \(i\) 张图的图像嵌入，\(\mathbf t_i\) 是第 \(i\) 句话的文本嵌入；第一项是“图找文”（相似度矩阵的每一行做一次 softmax），第二项是“文找图”（每一列）。\(1/\tau\) 用对数参数化来学，并截断在 100 以内；FIG 5 也这样做，从 \(1/\tau=5\) 开始，2000 步时升到约 13，之后在 14 左右。

SimCLR's encoder \(f_\theta\) (a ResNet in the paper) is followed by a small MLP projection head \(g\). The loss acts on \(\mathbf z\), but downstream tasks use the pre-projection \(\mathbf h\): the head actively discards augmentation-related information (such as colour) that \(\mathbf h\) still keeps, so linear classifiers work better on \(\mathbf h\). The standard way to judge a representation is the linear probe: freeze the encoder and train only a linear classifier on \(\mathbf h\) (Lesson 21.1 compares it with full fine-tuning). Our small MLP has no separate head and is evaluated by nearest neighbours directly on \(\mathbf z\). In the CLIP formula \(\mathbf u_i\) is the image embedding of image \(i\) and \(\mathbf t_i\) the text embedding of sentence \(i\); the first term is "image finds text" (a softmax over each row of the similarity matrix), the second "text finds image" (over each column). \(1/\tau\) is learned through a log parametrization and clipped at 100. FIG 5 does the same, starting from \(1/\tau=5\), reaching about 13 by step 2000 and about 14 later.

机器人的数据天然是视频，时间本身就提供了正样本。TCN（时间对比网络，Sermanet 等，2018）用多台相机同时拍同一个动作：同一时刻、不同相机的两帧是正样本，同一相机、不同时刻的帧是负样本。前者逼表示忽略视角和外观，后者逼它记住随时间变化的东西，也就是机器人和物体的状态。只用单台相机、把“相邻时刻”当正样本也行，但有个陷阱：同一段视频里每一帧都共享光照和背景，网络可以靠“这是哪段视频”配对，而不必在不同的视频之间对齐状态。

下图是一个夹爪在 10×10 的俯视相机里从随机起点移到目标（右上方）的 20 段测试演示，每段视频有自己的光照梯度、增益和一个固定的干扰方块。训练时每步取 10 段新演示、每段 3 个时刻：选“多视角”时，正样本是同一时刻、另一台相机（光照与干扰物都不同）的画面；选“单视角时间”时，正样本是同一相机的下一帧。评估方法是跨演示检索：对每一帧，在其他演示里找表示最近的帧，看两帧的夹爪位置差多少像素。这正是“用别人的演示来判断我现在进行到哪一步”时需要的能力。

Robot data is naturally video, and time itself supplies positives. TCN (Time-Contrastive Networks, Sermanet et al., 2018) films the same action with several cameras at once: two frames at the same instant from different cameras are positives, frames from the same camera at different times are negatives. The first forces the representation to ignore viewpoint and appearance; the second forces it to keep what changes over time, namely the state of the robot and the objects. A single camera with "neighbouring instants" as positives also works, but there is a trap: every frame of one video shares its lighting and background, so the network can pair frames by "which video is this" without aligning states across different videos.

The figure shows 20 test demonstrations of a gripper moving, in a 10×10 top-down camera, from a random start to a goal (upper right). Each video has its own lighting gradient, gain and a fixed distractor box. Each training step takes 10 new demonstrations and 3 instants from each: with "multi-view", the positive is the frame at the same instant from another camera (different lighting and distractor); with "single-view time", the positive is the next frame from the same camera. The evaluation is cross-demonstration retrieval: for each frame, find the closest frame in representation space among the other demonstrations and measure how many pixels apart the two gripper positions are. That is exactly the ability needed to "tell how far along I am by looking at someone else's demonstration".

同一个思想在机器人学习里反复出现。R3M（Nair 等，2022）在人类第一视角视频数据集 Ego4D 上，用时间对比、视频与语言对齐和 L1 稀疏正则训练一个 ResNet，然后冻结它，供下游的行为克隆（22.1 节）在少量演示上学操作策略。VIP（Ma 等，2023）同样只用 Ego4D、不需要动作标签，把训练目标写成一个隐式的、以目标为条件的价值函数，得到的表示里“到目标图像的距离”可以直接当作奖励，正是 FIG 6 右上角那条曲线的思路。CLIP 则把语言带进了机器人：CLIPort（Shridhar 等，2021）用 CLIP 的特征把语言指令和相机画面对应起来，基于 CLIP 式训练的开放词汇检测器（如 OWL-ViT）让机器人按名字找物体。今天的视觉-语言-动作模型也沿用对比预训练的图像编码器：OpenVLA 把 SigLIP 和 DINOv2 的特征拼在一起，π0 建在 PaliGemma 上，其视觉编码器也是 SigLIP。SigLIP（Zhai 等，2023）把 CLIP 的 softmax 换成对每个图文对独立的 sigmoid 二分类，不再需要整批归一化。

The same idea recurs throughout robot learning. R3M (Nair et al., 2022) trains a ResNet on the egocentric human video dataset Ego4D with time-contrastive learning, video–language alignment and an L1 sparsity penalty, then freezes it for downstream behaviour cloning (Lesson 22.1) of manipulation policies from a small number of demonstrations. VIP (Ma et al., 2023), also trained on Ego4D without action labels, casts the objective as an implicit goal-conditioned value function, so that "distance to the goal image" in its representation can be used directly as a reward: the idea behind the curve at the top right of FIG 6. CLIP brought language into robotics: CLIPort (Shridhar et al., 2021) uses CLIP features to ground language instructions in what the camera sees, and open-vocabulary detectors built on CLIP-style training (such as OWL-ViT) let a robot find objects by name. Today's vision-language-action models still use contrastively pre-trained image encoders: OpenVLA concatenates SigLIP and DINOv2 features, and π0 is built on PaliGemma, whose vision encoder is also SigLIP. SigLIP (Zhai et al., 2023) replaces CLIP's softmax by an independent sigmoid classification of each image–text pair, so no normalization over the whole batch is needed.