预训练与微调（LoRA）Pre-training & Fine-tuning (LoRA)

机器人数据很贵。一条遥操作演示要一个人花上一两分钟，一台机器人一天也就采几百条；而 17.3 节那样的 Transformer 动辄上亿参数，从零开始训练需要的数据量远远超过一个实验室能采到的。另一方面，互联网上有数十亿张图片和句子，开放数据集里有上百万条来自几十种机器人的轨迹。预训练 + 微调把这两件事接起来：先在“广而杂”的大数据上学到通用的表示和技能（预训练），再用目标任务的“少而精”的数据做一次小幅修改（微调）。

18.3 节已经见过最简单的一种：冻结预训练好的编码器，只在它的特征上训练一个线性分类器（线性探针）。本节把问题问完整：什么时候冻结就够了？什么时候必须改动网络本身？改动之后，模型原来会的东西还在不在？一个 70 亿参数的视觉-语言-动作模型（VLA），要怎样才能在一张显卡上、用一两百条演示完成微调？答案的核心是 LoRA：不改原始权重，只学一个低秩的增量 \(\Delta W=BA\)。

Robot data is expensive. One teleoperated demonstration costs a person a minute or two, and a robot collects a few hundred in a day; yet a transformer like those of Lesson 17.3 easily has hundreds of millions of parameters, and training one from scratch needs far more data than a lab can collect. Meanwhile the internet holds billions of images and sentences, and open datasets hold millions of trajectories from dozens of robots. Pre-training + fine-tuning connects the two: first learn general representations and skills from broad, messy, big data (pre-training), then make a small modification with the target task's small, clean data (fine-tuning).

Lesson 18.3 already showed the simplest version: freeze a pre-trained encoder and train only a linear classifier on its features (a linear probe). This lesson asks the full question: when is freezing enough? When must the network itself change? After it changes, does the model still know what it knew before? And how can a 7-billion-parameter vision-language-action model (VLA) be fine-tuned on one GPU from one or two hundred demonstrations? The heart of the answer is LoRA: leave the original weights alone and learn only a low-rank increment \(\Delta W=BA\).

本节的玩具问题是一个真实的机器人问题：保持力矩。机械臂第 2 个关节要停在角度 \(q\) 不动，电机需要输出多大的力矩 \(\tau(q)\)？它由重力（8.1 节的 \(g(q)\)，形如 \(a\cos q+b\sin q\)，\(a,b\) 取决于负载和质心）、传感器零偏，以及传动系统里每转重复 4 次的力矩波动组成。同一型号的 32 台机器人负载、零偏、波动的相位都不一样，我们在仿真里为每台采了大量数据。

预训练的网络是一个“主干 + 多个头”的结构：关节角以 \((\cos q,\sin q)\) 的形式输入（转动关节的常规编码），经过两层 32 个单元的 tanh 隐藏层，再到一个 6 维的表示 \(\hl{phi}{\boldsymbol\phi(q)}\)；每台机器人各有一个线性的头，\(\tau_k(q)\approx\mathbf w_k^\top\boldsymbol\phi(q)+b_k\)。为了让 32 个头都满意，主干必须学会这个机器人家族共有的结构。页面打开时这次预训练就在后台跑完了：3000 步，32 台机器人的平均误差降到约 0.014 N·m。

现在来了第 33 台机器人，我们只测了 12 个角度。最省事的做法是冻结主干，只为它拟合一个新的头 \(\hl{head}{\mathbf w}\)：这就是线性探针（linear probe），也就是 1.13 节的最小二乘，基函数换成了 \(\phi_1(q),\dots,\phi_6(q)\)。

This lesson's toy problem is a real robot problem: the holding torque. For joint 2 of an arm to stay still at angle \(q\), how much torque \(\tau(q)\) must its motor produce? It consists of gravity (the \(g(q)\) of Lesson 8.1, of the form \(a\cos q+b\sin q\), with \(a,b\) set by the payload and the centre of mass), a sensor offset, and a torque ripple from the drive train that repeats 4 times per revolution. The 32 robots of one model all differ in payload, offset and ripple phase, and we have collected plenty of simulated data for each.

The pre-trained network has a "trunk + many heads" structure: the joint angle enters as \((\cos q,\sin q)\) (the usual encoding of a revolute joint), passes through two tanh hidden layers of 32 units, and arrives at a 6-dimensional representation \(\hl{phi}{\boldsymbol\phi(q)}\); each robot has its own linear head, \(\tau_k(q)\approx\mathbf w_k^\top\boldsymbol\phi(q)+b_k\). To satisfy all 32 heads, the trunk must learn the structure the robot family has in common. This pre-training already ran in the background when the page opened: 3000 steps, after which the mean error over the 32 robots is about 0.014 N·m.

Now a 33rd robot arrives, and we have measured it at only 12 angles. The cheapest option is to freeze the trunk and fit just a new head \(\hl{head}{\mathbf w}\) for it: a linear probe, which is the least squares of Lesson 1.13 with the basis functions \(\phi_1(q),\dots,\phi_6(q)\).

给新机器人准备两种情况。同一家族：只是换了负载，曲线仍是重力 + 零偏 + 波动。新的效应：同样的机器人，但装了一个限位弹簧，\(q\) 超过 1 rad 后多出一段约 2 N·m/rad 的线性回弹力矩，这是预训练数据里从没有过的形状。每种情况采 \(N\) 个带噪声的测量（噪声标准差 0.05 N·m），四种方法同时训练：

• 从零训练：同样结构的网络，随机初始化，所有参数都训练。
• 线性探针：主干冻结，新头用岭回归一次解出（凸问题，不需要迭代，所以它的曲线一开始就是终点）。
• 全量微调：从预训练权重出发，所有参数都训练；新头先用线性探针的解初始化（这个做法叫 LP-FT，第 4 节解释原因；LoRA 的新头也这样初始化）。
• LoRA：所有预训练权重冻结；第二个隐藏层的 32×32 权重矩阵加上一个秩为 \(r\) 的增量 \(\frac{\alpha}{r}BA\)，只训练 \(B\)（32×\(r\)）、\(A\)（\(r\)×32）和新头。\(B\) 初始化为零，所以训练开始时模型和预训练模型一模一样。

四种方法用同样的 Adam 优化器、同样的批次（每步最多 32 个样本），训练 1000 步后自动停下。学习率按各自的习惯取：从零训练和 LoRA 用 0.003，全量微调用小一些的 0.001。

The new robot comes in two versions. Same family: only the payload changed, so the curve is still gravity + offset + ripple. New effect: the same robot fitted with a hard-stop spring, which adds a linear restoring torque of about 2 N·m/rad beyond \(q=1\) rad, a shape that never appeared in the pre-training data. For each version we take \(N\) noisy measurements (noise standard deviation 0.05 N·m) and train four methods side by side:

• From scratch: a network of the same shape, randomly initialized, all parameters trained.
• Linear probe: trunk frozen, the new head solved in one shot by ridge regression (a convex problem with no iterations, so its curve starts at its final value).
• Full fine-tuning: start from the pre-trained weights and train all parameters; the new head is first initialized with the linear-probe solution (this recipe is called LP-FT; Section 4 explains why; LoRA's new head is initialized the same way).
• LoRA: all pre-trained weights frozen; the second hidden layer's 32×32 weight matrix gets a rank-\(r\) increment \(\frac{\alpha}{r}BA\), and only \(B\) (32×\(r\)), \(A\) (\(r\)×32) and the new head are trained. \(B\) starts at zero, so at the start of training the model is exactly the pre-trained one.

All four use the same Adam optimizer and the same batches (at most 32 samples per step), and stop automatically after 1000 steps. Learning rates follow common practice for each: 0.003 for scratch and LoRA, a smaller 0.001 for full fine-tuning.

几个值得注意的地方。(1) 线性探针只有 7 个参数，对家族内的任务它几乎不可能过拟合；但在“新的效应”下，无论给多少数据，它都停在约 0.15 N·m：这是偏差，冻结的特征表达不了弹簧的折角（16.2 节的偏差–方差）。(2) 全量微调和 LoRA 在“同一家族”、\(N=16\) 时比它们出发时的探针更差：看右上的红线和绿线：它们从探针的误差出发，随着训练把噪声也拟合进去，误差一路升高（全量微调在第 200 步约 0.11，到 1000 步约 0.21）。这是微调最常见的失败方式，对策是早停、更小的学习率或者干脆只做探针。(3) 在“新的效应”下，默认的 \(N=64\) 里从零训练（约 0.07 N·m）已经比探针好，但仍不如两种微调（约 0.05 N·m）：预训练学到的重力和波动形状，从零训练要用更多数据才能自己找到。

A few things to notice. (1) The linear probe has only 7 parameters and can hardly overfit an in-family task; but with "new effect" it stalls at about 0.15 N·m however much data it gets. That is bias: the frozen features cannot express the spring's kink (the bias–variance trade-off of Lesson 16.2). (2) With "same family" and \(N=16\), full fine-tuning and LoRA end up worse than the probe they started from: in the top-right chart the red and green lines start at the probe's error and climb as training fits the noise as well (full fine-tuning is at about 0.11 by step 200 and about 0.21 by step 1000). This is the most common way fine-tuning fails, and the remedies are early stopping, a smaller learning rate, or just probing. (3) With "new effect" at the default \(N=64\), scratch (about 0.07 N·m) already beats the probe, but not the two fine-tuned models (about 0.05 N·m): the gravity and ripple shapes that pre-training learned take scratch training more data to discover on its own.

一次训练的结果受随机性影响，下面把 \(N\) 从 8 扫到 256，每个 \(N\) 用 3 组不同的数据和初始化各训练 1000 步，画出测试误差的几何平均。对数–对数坐标下，曲线之间的水平距离就是“省了多少倍的数据”。

A single run depends on chance, so below \(N\) is swept from 8 to 256, and for each \(N\) every method is trained for 1000 steps on 3 different datasets and initializations; the plot shows the geometric mean of the test error. On log–log axes the horizontal distance between curves is "how many times less data" one method needs.

一个基础模型的价值在于它会很多东西。在新机器人上微调之后，那 32 台老机器人它还能用吗？我们保留每台老机器人原来的头不动，把它们接在微调后的主干上，再测一次误差（下文称“旧机器人误差”）。预训练模型本身的旧机器人误差约 0.014 N·m。如果微调把它大幅抬高，就叫灾难性遗忘（catastrophic forgetting，McCloskey 与 Cohen 1989）：神经网络的知识分布在所有共享的权重里，为新任务改动这些权重，就会顺带改动旧任务的答案。

三种常见的对策：(1) 回放（rehearsal / replay）：微调时混入一部分旧数据，损失变成 \(L+\hl{replay}{\beta}\,L_0\)（\(L\) 是新任务的损失，\(L_0\) 是旧任务的损失）；机器人领域常说的“联合微调”（co-fine-tuning，RT-2 用网络数据和机器人数据一起微调）就是这一类。(2) 把参数拉回预训练值：加正则项 \(\hl{l2sp}{\lambda}\|\boldsymbol\theta-\boldsymbol\theta_0\|^2\)（L2-SP，Li 等 2018）；EWC（Kirkpatrick 等 2017）是它的加权版本，对旧任务越重要的参数拉得越紧。(3) 不动原始权重：LoRA 和各种适配器把改动放在额外的参数里，随时可以拿掉。

A foundation model is valuable because it knows many things. After fine-tuning on the new robot, can it still serve the 32 old ones? We keep each old robot's original head unchanged, attach it to the fine-tuned trunk and measure the error again (the "old-robot error" below). The pre-trained model itself has an old-robot error of about 0.014 N·m. When fine-tuning pushes it up a lot, that is catastrophic forgetting (McCloskey and Cohen 1989): a neural network's knowledge is spread over all the shared weights, so changing them for the new task also changes the answers to the old ones.

Three common remedies: (1) Replay (rehearsal): mix some old data into fine-tuning, so the loss becomes \(L+\hl{replay}{\beta}\,L_0\) (\(L\) the new task's loss, \(L_0\) the old tasks'); what robotics calls co-fine-tuning (RT-2 fine-tunes on web data and robot data together) is of this kind. (2) Pull the parameters back towards their pre-trained values: add a penalty \(\hl{l2sp}{\lambda}\|\boldsymbol\theta-\boldsymbol\theta_0\|^2\) (L2-SP, Li et al. 2018); EWC (Kirkpatrick et al. 2017) is a weighted version that holds parameters tighter the more they matter to the old tasks. (3) Do not touch the original weights: LoRA and the various adapters put the change into extra parameters that can be removed at any time.

LP-FT 的道理（Kumar 等 2022）：随机初始化的头一开始给出的是错得离谱的预测，它产生的大梯度会一路传回主干，在头学好之前就把预训练特征扭曲了；先用线性探针把头放到合理的位置，再一起微调，主干只需要做小的修正。在 FIG 4 里，这一个开关就让遗忘差了约四倍，而新机器人误差几乎不变（约 0.05 N·m）。

回放在这个玩具里比 L2-SP 划算：L2-SP 把所有参数一视同仁地往回拉，要压住遗忘就得把新任务也一起压住（\(\lambda=1\) 时旧机器人误差约 0.04，新机器人误差却退到约 0.12）；回放直接告诉网络“这些答案不许变”，网络会把改动放到旧任务不在乎的方向上。代价是你得一直保留、并且能用到旧数据。最后，把 LoRA 适配器“关掉”试试：旧机器人误差立刻回到 0.014，因为原始权重一个都没动。一个基础模型加上很多个可插拔的小适配器（每台机器人、每个任务一个），是 LoRA 在部署上最实际的好处。

Why LP-FT works (Kumar et al. 2022): a randomly initialized head starts with wildly wrong predictions, and its large gradients flow all the way back into the trunk, distorting the pre-trained features before the head has learned anything. Putting the head in a sensible place with a linear probe first, then fine-tuning everything, leaves the trunk only small corrections to make. In FIG 4 this one switch changes forgetting by a factor of about four while the new-robot error barely moves (about 0.05 N·m).

In this toy, replay is a better deal than L2-SP: L2-SP pulls every parameter back alike, so holding down forgetting also holds down the new task (at \(\lambda=1\) the old-robot error is about 0.04, but the new-robot error falls back to about 0.12); replay tells the network directly "these answers must not change", and the network moves in directions the old tasks do not care about. The price is that you must keep, and be allowed to use, the old data. Finally, try switching the LoRA adapter "off": the old-robot error returns at once to 0.014, because not a single original weight was touched. One foundation model plus many small pluggable adapters (one per robot or per task) is LoRA's most practical advantage in deployment.

LoRA（Hu 等 2021）基于一个经验观察：预训练模型在下游任务上需要的改动，其“内在维度”很低（Aghajanyan 等 2021 在语言模型上测过：在一个随机的低维子空间里优化几百到几千个参数，就能达到全量微调的大部分效果）。那就直接把权重矩阵的改动限制成低秩：

LoRA (Hu et al. 2021) rests on an empirical observation: the change a pre-trained model needs for a downstream task has a low "intrinsic dimension" (Aghajanyan et al. 2021 measured it for language models: optimizing a few hundred to a few thousand parameters in a random low-dimensional subspace recovers most of the effect of full fine-tuning). So restrict the change of each weight matrix to low rank directly:

训练的参数从 \(dk\) 个降到 \(r(d+k)\) 个。下面的图先用全量微调在“新的效应”、\(N=128\) 上训练 1000 步，把第二个隐藏层的改动 \(\Delta W=W-W_0\) 拿出来做 1.12 节的 SVD：如果它的奇异值 \(\hl{sigma}{\sigma_i}\) 很快衰减，那么一个低秩矩阵就能代替它。同时训练一个 \(r=2\) 的 LoRA，看它的 \(\Delta W\) 长什么样。

The trained parameters drop from \(dk\) to \(r(d+k)\). The figure below first runs full fine-tuning on "new effect" with \(N=128\) for 1000 steps, takes the change of the second hidden layer \(\Delta W=W-W_0\), and applies the SVD of Lesson 1.12: if its singular values \(\hl{sigma}{\sigma_i}\) decay quickly, a low-rank matrix can stand in for it. It also trains a LoRA with \(r=2\), to see what its \(\Delta W\) looks like.

第二个任务背后是一条简单而重要的事实：对一个样本，线性层 \(\mathbf z=W\mathbf h\) 的梯度是外积 \(\partial L/\partial W=\boldsymbol\delta\,\mathbf h^\top\)（16.3 节），秩为 1；一个批次 \(n\) 个样本的梯度是 \(n\) 个外积之和，秩不超过 \(n\)。微调的数据少、步数少、而且每步的梯度方向高度相关，累积起来的改动自然集中在少数几个方向上。LoRA 的 \(B\) 和 \(A\) 就是让优化器直接去找这几个方向。

LoRA 还有一个好处：训练完可以合并，\(W=W_0+\frac{\alpha}{r}BA\) 只是一个同样大小的矩阵，推理时的计算量和延迟与原模型完全相同（读数里的“合并误差”是合并前后输出的最大差，只剩浮点舍入）。想换任务时减去 \(\frac{\alpha}{r}BA\)、加上另一个即可。

Behind the second task is a simple but important fact: for one sample, the gradient of a linear layer \(\mathbf z=W\mathbf h\) is the outer product \(\partial L/\partial W=\boldsymbol\delta\,\mathbf h^\top\) (Lesson 16.3), of rank 1; the gradient of a batch of \(n\) samples is a sum of \(n\) outer products, of rank at most \(n\). Fine-tuning has little data and few steps, and the gradient directions of successive steps are highly correlated, so the accumulated change naturally concentrates in a few directions. LoRA's \(B\) and \(A\) let the optimizer look for those directions directly.

LoRA has one more advantage: after training it can be merged. \(W=W_0+\frac{\alpha}{r}BA\) is just a matrix of the same size, so inference costs exactly the same compute and latency as the original model (the readout's "merge error" is the largest difference in output before and after merging: only floating-point rounding). To switch tasks, subtract \(\frac{\alpha}{r}BA\) and add another one.

LoRA 让需要训练的参数少了几百倍，但冻结的底座仍然要整个放进显存：70 亿参数用 16 位存就是 14 GB。全量微调更贵：混合精度的 Adam 每个参数要存 16 字节（16 位权重和梯度各 2 字节，32 位的权重主副本 4 字节，Adam 的两个矩估计 8 字节，ZeRO 论文的估算），70 亿参数就是 112 GB，还没算激活值。QLoRA（Dettmers 等 2023）把冻结的底座量化成 4 位，LoRA 的参数仍用 16 位训练，梯度穿过反量化后的权重传给 \(B\) 和 \(A\)。

最简单的量化是分块 absmax：把权重分成小块，每块记下最大绝对值 \(c\)，再把 \(w/c\in[-1,1]\) 四舍五入到 \(2^{b}-1\) 个等距的格点之一，只存格点编号（\(b\) 位）和每块一个 \(c\)。下面对我们的预训练网络这样做（每块是权重矩阵的一行，第一层则是一列；大多数块有 32 个数），看量化让老机器人的误差变大了多少，再在量化后的底座上训练 LoRA。

LoRA cuts the parameters that need training by a factor of hundreds, but the frozen base must still fit in GPU memory in full: 7 billion parameters in 16 bits is 14 GB. Full fine-tuning is far more expensive: mixed-precision Adam stores 16 bytes per parameter (2 bytes each for the 16-bit weight and gradient, 4 for a 32-bit master copy of the weight, 8 for Adam's two moment estimates; the estimate of the ZeRO paper), which for 7 billion parameters is 112 GB before counting activations. QLoRA (Dettmers et al. 2023) quantizes the frozen base to 4 bits while the LoRA parameters are still trained in 16 bits, with gradients flowing through the de-quantized weights into \(B\) and \(A\).

The simplest quantization is block-wise absmax: split the weights into small blocks, record each block's largest magnitude \(c\), round \(w/c\in[-1,1]\) to one of \(2^{b}-1\) evenly spaced levels, and store only the level index (\(b\) bits) plus one \(c\) per block. Below we do this to our pre-trained network (one block = one row of a weight matrix, or one column in the first layer; most blocks hold 32 numbers), see how much the quantization raises the old robots' error, and then train a LoRA on top of the quantized base.

注意两个现象。第一，在 2 位底座上，LoRA 训练开始时的误差（约 0.20 N·m）比 16 位底座（约 0.15）差，但训练会把量化误差和新任务一起“修”掉：低秩增量加一个新头，足够补偿量化带来的那部分偏差。QLoRA 论文报告，用 4 位底座 + LoRA 微调，能达到 16 位全量微调的效果。第二，我们这个只有 1357 个参数的小网络对量化非常敏感（4 位时老机器人误差约 0.1 N·m，是全精度的 7 倍）；大模型的权重多、冗余大，对 4 位量化宽容得多。QLoRA 的 4 位格点也不是等距的，而是按正态分布的分位数放置（NF4），因为预训练权重大致服从零均值正态分布，靠近 0 的地方格点更密；它还把每块 64 个权重的常数 \(c\) 再量化一次（双重量化），并用分页优化器应对显存峰值。

Notice two things. First, on the 2-bit base LoRA starts from a worse error than on the 16-bit base (about 0.20 versus 0.15 N·m), but training "repairs" the quantization error along with the new task: the low-rank increment plus a new head are enough to compensate for the bias the quantization introduced. The QLoRA paper reports that a 4-bit base plus LoRA matches 16-bit full fine-tuning. Second, our tiny 1357-parameter network is very sensitive to quantization (at 4 bits the old-robot error is about 0.1 N·m, 7 times the full-precision value); big models, with many more and much more redundant weights, tolerate 4 bits far better. QLoRA's 4-bit levels are also not evenly spaced but placed at the quantiles of a normal distribution (NF4), because pre-trained weights are roughly zero-mean normal and the levels should be denser near 0; it also quantizes the per-block constants \(c\) (one per 64 weights) a second time (double quantization) and uses paged optimizers to absorb memory spikes.

记预训练得到的主干参数为 \(\boldsymbol\psi_0\)，新任务数据为 \(\{(x_i,y_i)\}_{i=1}^N\)，新头为 \(\mathbf w\)，模型 \(\hat y=\mathbf w^\top\boldsymbol\phi_{\boldsymbol\psi}(x)\)。四种做法只是“哪些变量可以动、从哪里出发”不同：

Let \(\boldsymbol\psi_0\) be the pre-trained trunk parameters, \(\{(x_i,y_i)\}_{i=1}^N\) the new task's data, \(\mathbf w\) the new head and \(\hat y=\mathbf w^\top\boldsymbol\phi_{\boldsymbol\psi}(x)\) the model. The four approaches differ only in which variables may move and where they start:

四行依次是：线性探针（LP）；全量微调，从探针的解出发（LP-FT）；加在全量微调上的三种防遗忘项，L2-SP、EWC 和回放；LoRA。线性探针里 \(\Phi\) 的第 \(i\) 行是 \(\boldsymbol\phi_{\boldsymbol\psi_0}(x_i)^\top\)（末尾补一个 1 作偏置），这就是 1.13 节的正规方程加岭项（FIG 1 和 FIG 2 用 \(\lambda=0.01\)，偏置不受惩罚）。第三行的中间一项是 EWC：\(F_i\) 是旧任务上 Fisher 信息矩阵的对角元，粗略地说是“损失对第 \(i\) 个参数有多敏感”，所以 EWC 是“按重要性加权的 L2-SP”。回放项 \(L_0\) 在 FIG 4 里是 32 台老机器人通过各自冻结的头的均方误差，每步随机取 16 个角度。悬停这些公式，FIG 2 和 FIG 4 里对应的曲线会加粗。

The four lines are, in order: the linear probe (LP); full fine-tuning, started from the probe's solution (LP-FT); three anti-forgetting terms added to full fine-tuning, L2-SP, EWC and replay; and LoRA. In the linear probe, row \(i\) of \(\Phi\) is \(\boldsymbol\phi_{\boldsymbol\psi_0}(x_i)^\top\) (with a 1 appended for the bias): the normal equations of Lesson 1.13 plus a ridge term (FIG 1 and FIG 2 use \(\lambda=0.01\), with the bias not penalized). The middle term of the third line is EWC: \(F_i\) is the diagonal of the Fisher information matrix on the old tasks, roughly "how sensitive the old loss is to parameter \(i\)", so EWC is "L2-SP weighted by importance". The replay term \(L_0\) in FIG 4 is the mean squared error of the 32 old robots through their frozen heads, at 16 random angles per step. Hover these formulas and the corresponding curves in FIG 2 and FIG 4 thicken.

令 \(G=\partial L/\partial W\) 为合并后权重 \(W=W_0+sBA\)（\(s=\alpha/r\)）上的普通梯度，也就是全量微调会用到的那个梯度。由链式法则：

Let \(G=\partial L/\partial W\) be the ordinary gradient on the merged weight \(W=W_0+sBA\) (\(s=\alpha/r\)), the very gradient full fine-tuning would use. By the chain rule:

页面的代码正是这样做的：用 nn.js 在合并后的网络上反传得到 \(G\)，再乘上 \(A^\top\) 或 \(B^\top\)。从这两个式子能读出初始化的讲究：如果 \(A\) 和 \(B\) 都从零开始，两个梯度都是零，永远动不了；如果都随机，训练一开始模型就被一个随机的 \(sBA\) 扰乱了。所以 LoRA 让 \(B=0\)、\(A\) 随机：起点就是预训练模型，而 \(\partial L/\partial B=sGA^\top\neq0\)，第一步就能动起来。缩放 \(s=\alpha/r\) 让“改变秩”和“改变学习率”大致解耦：\(r\) 变大时每一列的贡献按 \(1/r\) 缩小，LoRA 论文因此把 \(\alpha\) 固定为第一个尝试的 \(r\)，之后不再调它（我们取 \(\alpha=2r\)）。

参数个数。一个 \(d\times k\) 的矩阵：全量 \(dk\)，LoRA \(r(d+k)\)。\(d=k=4096\)、\(r=8\) 时是 65,536 对 16,777,216，约 0.39%。我们的玩具：全量微调 1357 个参数（\(2\cdot32+32\) + \(32\cdot32+32\) + \(32\cdot6+6\) + 新头 7），LoRA \(r=2\) 是 \(2\cdot2\cdot32=128\) 加新头 7，共 135 个。

适配器（adapter，Houlsby 等 2019）是更早的做法：在每个 Transformer 子层后面插入一个小瓶颈 \(\mathbf h\leftarrow\mathbf h+W_{\uparrow}\,\sigma(W_{\downarrow}\mathbf h)\)，\(W_{\downarrow}\in\mathbb R^{m\times d}\)，\(m\ll d\)，只训练这些小矩阵。参数同样很少，但中间有非线性 \(\sigma\)，没法并进原权重，推理时多出串行的计算。提示微调 / 前缀微调则只学几个附加在输入序列（或每层键值）前面的向量，参数更少，表达力也更弱。

The page's code does exactly this: it backpropagates through the merged network with nn.js to get \(G\), then multiplies by \(A^\top\) or \(B^\top\). These two formulas also explain the initialization. If \(A\) and \(B\) both start at zero, both gradients are zero and nothing ever moves; if both are random, the model is scrambled by a random \(sBA\) before training begins. So LoRA sets \(B=0\) and \(A\) random: the starting point is the pre-trained model, yet \(\partial L/\partial B=sGA^\top\neq0\), so the first step already moves. The scaling \(s=\alpha/r\) roughly decouples "changing the rank" from "changing the learning rate": as \(r\) grows, each column's contribution shrinks as \(1/r\), which is why the LoRA paper fixes \(\alpha\) to the first \(r\) it tries and does not tune it afterwards (we use \(\alpha=2r\)).

Parameter counts. For a \(d\times k\) matrix: \(dk\) in full, \(r(d+k)\) with LoRA. With \(d=k=4096\) and \(r=8\) that is 65,536 against 16,777,216, about 0.39%. Our toy: full fine-tuning trains 1357 parameters (\(2\cdot32+32\) + \(32\cdot32+32\) + \(32\cdot6+6\) + the new head's 7), LoRA with \(r=2\) trains \(2\cdot2\cdot32=128\) plus the new head's 7, 135 in all.

Adapters (Houlsby et al. 2019) came earlier: insert a small bottleneck \(\mathbf h\leftarrow\mathbf h+W_{\uparrow}\,\sigma(W_{\downarrow}\mathbf h)\) after each transformer sub-layer, with \(W_{\downarrow}\in\mathbb R^{m\times d}\) and \(m\ll d\), and train only these small matrices. They also have few parameters, but the nonlinearity \(\sigma\) in the middle means they cannot be folded into the original weights, so inference gains extra sequential computation. Prompt tuning / prefix tuning learn only a few vectors prepended to the input sequence (or to each layer's keys and values): even fewer parameters, and less expressive power.

FIG 5 的 \(\Delta W_r\) 用的是 SVD 截断 \(\Delta W_r=\sum_{i\le r}\sigma_i\mathbf u_i\mathbf v_i^\top\)。Eckart–Young 定理（1.12 节）说它是所有秩不超过 \(r\) 的矩阵里离 \(\Delta W\) 最近的（Frobenius 范数下），误差恰好是 \(\|\Delta W-\Delta W_r\|_F^2=\sum_{i>r}\sigma_i^2\)，读数里的能量比例就是 \(\sum_{i\le r}\sigma_i^2/\sum_i\sigma_i^2\)。LoRA 并不先算全量微调再截断，而是直接在秩 \(r\) 的矩阵里优化，学到的 \(\Delta W\) 也不必等于全量微调的截断：FIG 5 里 LoRA 的 \(BA\) 和全量微调的 \(\Delta W\) 长得并不一样，测试误差却差不多，因为能把新任务做好的改动不止一个。

The \(\Delta W_r\) of FIG 5 is the SVD truncation \(\Delta W_r=\sum_{i\le r}\sigma_i\mathbf u_i\mathbf v_i^\top\). The Eckart–Young theorem (Lesson 1.12) says it is the closest matrix of rank at most \(r\) to \(\Delta W\) in the Frobenius norm, with error exactly \(\|\Delta W-\Delta W_r\|_F^2=\sum_{i>r}\sigma_i^2\); the readout's energy fraction is \(\sum_{i\le r}\sigma_i^2/\sum_i\sigma_i^2\). LoRA does not compute a full fine-tune and truncate it; it optimizes directly over rank-\(r\) matrices, and the \(\Delta W\) it learns need not equal the truncated full fine-tune: in FIG 5 LoRA's \(BA\) looks different from full fine-tuning's \(\Delta W\), yet the test errors are about the same, because more than one change does the new task well.

机器人学习里的预训练大致经历了三个阶段。冻结的视觉编码器：在 ImageNet、人类视频（R3M）或大规模图像上自监督（MVP、VC-1）预训练一个编码器，冻结后在它的特征上训练策略，就是 18.3 节和 18.4 节讲过的“线性探针”思路的延伸。从视觉-语言模型出发：RT-2（Brohan 等 2023）把动作写成文本 token（17.5 节），在大规模视觉-语言模型上用机器人数据微调；它采用联合微调，把原来的网络视觉问答数据和机器人轨迹混在一起训练，正是 FIG 4 的回放，目的是不忘掉从网络上学到的物体和概念。在跨本体机器人数据上预训练的通用策略：Octo（Octo Model Team 2024）在 Open X-Embodiment 的 80 万条轨迹上预训练，论文报告能在普通的消费级显卡上用几个小时把它微调到新的传感器输入和动作空间；OpenVLA（Kim 等 2024）是一个 70 亿参数的 VLA，在 97 万条真实机器人演示上训练。

OpenVLA 的论文给出了一组很能说明问题的数字。在每个任务只有 10 到 150 条演示的 Franka 机械臂任务上，全量微调需要 8 张 A100 每个任务训练 5 到 15 小时；LoRA（秩 32）只训练 1.4% 的参数，平均成功率 68.2%，与全量微调的 69.7% 相当，只用一张 A100 训练 10 到 15 小时；作者还发现秩的大小对成功率影响很小。部署时，4 位量化推理的成功率与 16 位推理相当，显存不到一半（8 位量化反而变差，作者归因于推理速度变慢）。π0（Black 等 2024）则沿用了语言模型“预训练 + 后训练”的说法：先在大量多机器人数据上预训练，再用精心筛选的高质量数据把它微调到叠衣服这类复杂的灵巧任务上。

Pre-training in robot learning has gone through roughly three stages. Frozen visual encoders: pre-train an encoder on ImageNet, on human video (R3M) or self-supervised on large image collections (MVP, VC-1), freeze it and train a policy on its features: an extension of the "linear probe" idea of Lesson 18.3 and Lesson 18.4. Starting from a vision-language model: RT-2 (Brohan et al. 2023) writes actions as text tokens (Lesson 17.5) and fine-tunes large vision-language models on robot data; it co-fine-tunes, mixing the original web visual-question-answering data with robot trajectories, which is exactly the replay of FIG 4, so as not to forget the objects and concepts learned from the web. Generalist policies pre-trained on cross-embodiment robot data: Octo (Octo Model Team 2024) is pre-trained on 800k trajectories from Open X-Embodiment, and its paper reports fine-tuning to new sensor inputs and action spaces within a few hours on standard consumer GPUs; OpenVLA (Kim et al. 2024) is a 7-billion-parameter VLA trained on 970k real robot demonstrations.

The OpenVLA paper reports a telling set of numbers. On Franka-arm tasks with only 10 to 150 demonstrations each, full fine-tuning took 8 A100 GPUs for 5 to 15 hours per task; LoRA (rank 32) trained only 1.4% of the parameters and reached a 68.2% average success rate, on par with full fine-tuning's 69.7%, using a single A100 for 10 to 15 hours; the authors also found that the rank had little effect on success. For deployment, 4-bit quantized inference matched the success rate of 16-bit inference with less than half the GPU memory (8-bit was worse, which the authors attribute to slower inference). π0 (Black et al. 2024) borrows the language-model terms "pre-training + post-training": pre-train on a large multi-robot mixture, then fine-tune on carefully curated high-quality data for complex dexterous tasks such as folding laundry.