相机标定与手眼标定Camera & Hand–Eye Calibration
前面三节的所有公式都假设内参 \(K\)、畸变和外参是已知的，可没有哪台相机会告诉你这些数。标定就是用许多测量去估计这些未知数：拍一块已知尺寸的棋盘格，从每张图的单应里用 Zhang 方法线性地解出 \(K\)，再对整个重投影误差做 Levenberg–Marquardt 精化；相机装在机械臂上时，还要估计相机相对手的位姿 \(X\)，也就是解 \(AX=XB\)。本节把两个算法都在页面里真实地跑一遍，看它们何时精确、何时退化，以及怎样判断一次标定到底好不好。 Every formula of the last three lessons assumed that the intrinsics \(K\), the distortion and the extrinsics are known, yet no camera tells you these numbers. Calibration estimates them from many measurements: photograph a checkerboard of known size, solve for \(K\) linearly from the per-image homographies with Zhang's method, then refine the whole reprojection error by Levenberg–Marquardt; and when the camera sits on a robot arm, also estimate the camera's pose \(X\) relative to the hand, which means solving \(AX=XB\). Both algorithms run live in the page, so you can see when they are exact, when they degenerate, and how to tell whether a calibration is any good.
1为什么机器人需要它Why robots need this
10.1 到 10.3 节的每个公式，都默认焦距、主点、畸变系数和外参是已知的。现实里没有哪台相机会把这些数告诉你：出厂说明书给的是同型号的典型值，换一次镜头、拧一下对焦环、磕一下支架、机身升温几十度，它们都会变。数错的后果是系统性的：像素到射线的映射整体歪掉，之后的三角化、PnP、点云全带着同一个偏差，滤波也平均不掉。
机械臂上的相机还多一个问题。相机看到的杯子坐标在相机系里，而手臂要去的位置写在基座系里，中间隔着一个从来没人量过的刚体变换：相机相对手的位姿 \(X={}^{h}T_{c}\)。这个变换差几度、几毫米，抓取就差几厘米。
- 内参与畸变标定（Zhang 方法）：看一块已知尺寸的平面标定板，估计 \(K\) 与畸变系数。
- 手眼标定（\(AX=XB\)）：让手臂换几个姿态，每个姿态都拍到同一个固定目标，由此解出 \(X\)。
- 判断好坏：重投影误差小就够了吗？为什么少数几张相似的图能给出"很小的误差"和"很错的相机"？
这两个问题其实是同一个故事，也是本课程反复出现的套路：先用线性（闭式）方法得到一个粗糙的初值，再把真正的误差函数交给非线性最小二乘精化（6.9 节）。闭式解给你"落在正确的盆地里"，精化给你"统计上最优"。
Every formula in Lessons 10.1 to 10.3 takes the focal length, principal point, distortion coefficients and extrinsics as given. No camera tells you these numbers: the datasheet gives typical values for the model, and swapping a lens, turning the focus ring, bumping a bracket or heating the body by tens of degrees all change them. Wrong numbers cause systematic errors: the pixel-to-ray map is bent as a whole, and triangulation, PnP and point clouds downstream all inherit the same bias, which no filter can average away.
A camera on a robot has one more problem. The cup is seen in the camera frame, but the arm has to go to a place written in the base frame, and between them lies a rigid transform nobody has measured: the camera's pose relative to the hand, \(X={}^{h}T_{c}\). Be off by a few degrees or millimetres there and the grasp misses by centimetres.
- Intrinsic and distortion calibration (Zhang's method): look at a flat board of known size and estimate \(K\) and the distortion.
- Hand–eye calibration (\(AX=XB\)): move the arm to several poses, photograph the same fixed target each time, and solve for \(X\).
- Judging quality: is a small reprojection error enough? How can a few similar images give a "tiny error" and a "wrong camera" at once?
The two problems are one story, and a recurring one in this course: get a rough initial guess from a linear (closed-form) method, then hand the true error function to nonlinear least squares (Lesson 6.9). The closed form puts you in the right basin; the refinement makes you statistically optimal.
2直觉：一块棋盘，几个角度Intuition: one checkerboard, a few viewing angles
拿一块棋盘格，它的 54 个内角点在板上的坐标是已知的（9×6 个角点，方格 40 mm）。每拍一张图，就得到 54 对"板上坐标 ↔ 像素坐标"，由它们解出一个 3×3 的单应 \(H\)（10.5 节会深入讲；这里把它当作"平面的 DLT"：每对点给两个线性方程，用 SVD 解，和 10.2 节讲过的 DLT 是同一个方法）。
单应里同时藏着两样东西：棋盘相对相机的位姿，以及相机自己的 \(K\)。靠什么把它们分开？靠"棋盘是个刚性平面"这一个事实：旋转矩阵的两列必须正交且等长。这对每张图给出关于 \(K\) 的 2 个线性方程；\(K\) 有 4 到 5 个未知数，所以几张不同角度的图就够了。角度不同是关键：棋盘正对着相机时，透视"近大远小"的信息几乎没有，焦距和距离分不开；把它斜过来，焦距才从距离里"露"出来。
FIG 1 在页面里真实运行整套算法：左边是虚拟相机（青色视锥）和 \(n\) 块棋盘，右边是相机拍到的角点（带高斯噪声），下面是用前 \(k\) 张图估计的 \(\hat K\) 与真值的误差。拖动滑块改变视角的数量、倾斜度、噪声，也可以切到"全部正对相机"，看算法如何失败。
Take a checkerboard whose 54 inner corners have known coordinates on the board (9×6 corners, 40 mm squares). Every photo yields 54 pairs "board coordinate ↔ pixel", from which we solve a 3×3 homography \(H\) (Lesson 10.5 treats it in depth; here read it as "the DLT of a plane": every pair gives two linear equations, solved by SVD, the same DLT as in Lesson 10.2).
A homography hides two things at once: the board's pose relative to the camera, and the camera's own \(K\). What separates them is a single fact, that the board is a rigid plane: two columns of a rotation matrix must be orthogonal and of equal length. Each image therefore gives two linear equations on \(K\); \(K\) has 4 or 5 unknowns, so a few different views suffice. Different is the key word: a board facing the camera squarely carries almost no "near is bigger" perspective information, so focal length and distance cannot be told apart; tilt it and the focal length shows itself.
FIG 1 runs the whole algorithm for real: the left panel shows the virtual camera (cyan frustum) and \(n\) boards, the right panel the corners it sees (with Gaussian noise), and below is the error of \(\hat K\) estimated from the first \(k\) images. Drag the sliders for the number of views, tilt and noise, or switch to "all parallel" and watch the algorithm fail.
- 把噪声 \(\sigma\) 拉到 0，视角数 \(\ge3\)，模式用"多轴倾斜"：\(\hat K\) 与真值只差 \(10^{-9}\) 量级。无噪声时 Zhang 的闭式解是精确的，不是近似。Set the noise \(\sigma\) to 0 with \(\ge3\) views in "tilt, varied axes" mode: \(\hat K\) matches the truth to about \(10^{-9}\). Without noise Zhang's closed form is exact, not approximate.
- 把视角数调到 2，并在右上角把"轴倾斜 skew"切到"自由"：只有 4 个方程却要解 5 个内参，算法拒绝给出答案。再切回"skew = 0"，2 张图就够了。Set the views to 2 and switch skew to "free" at the top right: four equations cannot determine five intrinsics, and the algorithm refuses to answer. Switch back to "skew = 0" and two views suffice.
- 选"全部正对相机"，视角数 \(\ge6\)：约束健康度掉到 \(10^{-2}\) 以下，\(\hat K\) 变成垃圾。棋盘在自己的平面内转动、前后平移都没用，只有改变平面的朝向才增加方程。Choose "all parallel to the sensor" with \(\ge6\) views: the constraint health drops below \(10^{-2}\) and \(\hat K\) becomes garbage. Rolling the board in its plane or moving it back and forth does not help; only changing the plane's orientation adds equations.
- 把"方格尺寸误差"拉到 \(\pm10\%\) 以上：\(\hat K\) 完全不变，但估计的棋盘距离按同样比例错掉。尺度错误不会被重投影误差发现。Pull the "square-size error" beyond \(\pm10\%\): \(\hat K\) does not change at all, but the estimated board distance is off by the same ratio. A scale error is invisible to the reprojection error.
一张图 = 关于 \(K\) 的 2 个方程，与棋盘的位姿无关。\(K\) 有 \(5\) 个未知数（含 skew）时需要 \(\ge3\) 张朝向不同的图，\(4\) 个（skew = 0）时 2 张就够。"朝向不同"不是礼貌，是数学条件：朝向相同的图给出的是同一对方程。
One image = two equations on \(K\), independent of the board's pose. With \(5\) unknowns (skew included) you need \(\ge3\) views with different orientations; with \(4\) (skew = 0) two suffice. "Different orientations" is not politeness but a mathematical condition: views with the same orientation give the same pair of equations again.
3动手玩：重投影误差与非线性精化Play: reprojection error and nonlinear refinement
闭式解有两个缺点。第一，它最小化的是"代数误差"（方程的残差），不是图像里真正量到的东西；第二，它根本不知道畸变，可镜头是弯的。下一步是回到物理：把全部参数 \((K,\ k_1,k_2,\ \{R_i,\mathbf t_i\})\) 放在一起，最小化所有角点的重投影误差，也就是"模型预测的像素位置"与"检测到的像素位置"的距离平方和。这正是高斯噪声下的最大似然估计（4.6 节），用 Levenberg–Marquardt（6.6、6.9 节）从闭式解出发迭代。
FIG 2 里的虚拟镜头有桶形畸变。每次迭代都是一个真实的 LM 步（数值雅可比、Marquardt 阻尼 \(\lambda\,\mathrm{diag}(J^\top J)\)、接受或拒绝）。拖动"迭代"滑块可以倒带，右图的箭头是放大后的残差。读数中的"噪声下限"\(\sqrt2\,\sigma\sqrt{1-p/N}\)（二维距离的 RMS，见 5 节）是正确模型下 RMS 应当到达的值：它不会是 0，因为参数用掉了一部分噪声。
The closed form has two weaknesses. First, it minimizes an "algebraic error" (the residual of the equations), not what is actually measured in the image; second, it knows nothing about distortion, yet lenses are curved. The next step is to go back to the physics: put all parameters \((K,\ k_1,k_2,\ \{R_i,\mathbf t_i\})\) together and minimize the reprojection error of all corners, the sum of squared distances between "where the model predicts the pixel" and "where it was detected". That is exactly the maximum-likelihood estimate under Gaussian noise (Lesson 4.6), run by Levenberg–Marquardt (Lessons 6.6 and 6.9) from the closed-form start.
The virtual lens in FIG 2 has barrel distortion. Every iteration is a real LM step (numerical Jacobian, Marquardt damping \(\lambda\,\mathrm{diag}(J^\top J)\), accept or reject). Drag the "iteration" slider to rewind; the arrows on the right are the magnified residuals. The "noise floor" \(\sqrt2\,\sigma\sqrt{1-p/N}\) in the readout (the RMS of the 2-D distance, see Section 5) is the RMS a correct model should reach: it is not 0, because the parameters soak up a part of the noise.
- 把"迭代"滑块先拖到 0，再拖到最后：RMS 从闭式解的水平下降到噪声下限，箭头从有规律的径向模式变成无规律的短毛刺。Drag the "iteration" slider to 0 and then to the end: the RMS falls from the closed-form level to the noise floor, and the arrows change from a regular radial pattern into short random spikes.
- 用 \(k_1\le-0.2\)、\(\sigma\le0.5\)，先选"只有针孔"：RMS 停在噪声下限之上，残差箭头呈径向结构。再切到"针孔 + 径向畸变"，RMS 落回下限。残差有结构，就是模型错了。With \(k_1\le-0.2\) and \(\sigma\le0.5\), first choose "pinhole only": the RMS stalls above the noise floor and the residual arrows show radial structure. Then switch to "pinhole + radial distortion" and the RMS drops to the floor. Structure in the residuals means the model is wrong.
- 视角数调到 3，噪声 \(\sigma\ge1\)：RMS 仍贴着噪声下限（"看起来标定得很好"），但 \(f\) 的标准差超过 8 px。再加视角，标准差才收下来。小误差不等于好标定。Set the views to 3 and the noise to \(\sigma\ge1\): the RMS still hugs the noise floor ("looks well calibrated"), yet the standard deviation of \(f\) exceeds 8 px. Only more views shrink it. A small error is not a good calibration.
几何：让所有预测的角点尽量落在检测到的位置上。统计：噪声为各向同性高斯时，这是最大似然估计，协方差 \(\sigma^2(J^\top J)^{-1}\) 告诉你每个参数有多可信。优化：一个有 \(6+6n\) 个变量的非线性最小二乘，雅可比有分块稀疏结构（每个角点只依赖全局参数和它自己那张图的位姿），这正是 11.6 节光束法平差的雏形。
Geometry: make every predicted corner land on its detected position. Statistics: with isotropic Gaussian noise this is the maximum-likelihood estimate, and the covariance \(\sigma^2(J^\top J)^{-1}\) tells you how far to trust each parameter. Optimization: a nonlinear least-squares problem with \(6+6n\) variables whose Jacobian is block sparse (each corner depends only on the global parameters and its own image's pose), which is the seed of bundle adjustment in Lesson 11.6.
4第二、第三视角：手眼标定 \(AX=XB\)Second and third views: hand–eye calibration, \(AX=XB\)
相机在手上，目标固定在世界里The camera on the hand, the target fixed in the world
现在把相机装到机械臂腕部（eye-in-hand，眼在手上）。有三个变换在起作用：机器人的正运动学给出手的位姿 \({}^{b}T_{h}(\mathbf q)\)（7.3 节，已知）；在棋盘上做 PnP（11.4 节）给出棋盘在相机系里的位姿 \(P={}^{c}T_{t}\)（测量值，有噪声）；未知的是相机在手里的位姿 \(X={}^{h}T_{c}\)，棋盘在基座系的位姿 \({}^{b}T_{t}\) 也不知道，但它固定不动。
关键一步：对手臂的两个姿态 \(i,j\)，把同一个刚体运动分别从手和从相机的视角写出来。手走的是 \(A_{ij}={}^{h_i}T_{h_j}=({}^{b}T_{h,i})^{-1}\,{}^{b}T_{h,j}\)（机器人给出）；相机走的是 \(B_{ij}={}^{c_i}T_{c_j}=P_i\,P_j^{-1}\)（相机给出，棋盘在基座系的未知位姿在乘积里被消掉了）。手和相机是一个刚体，所以
Now mount the camera on the wrist (eye-in-hand). Three transforms are in play: the robot's forward kinematics gives the hand pose \({}^{b}T_{h}(\mathbf q)\) (Lesson 7.3, known); PnP on the checkerboard (Lesson 11.4) gives the board's pose in the camera frame, \(P={}^{c}T_{t}\) (measured, noisy); unknown is the camera's pose in the hand, \(X={}^{h}T_{c}\), and the board's pose in the base frame \({}^{b}T_{t}\) is unknown too, but it is fixed.
The key step: for two arm poses \(i,j\), write the same rigid motion from the hand's point of view and from the camera's. The hand moved by \(A_{ij}={}^{h_i}T_{h_j}=({}^{b}T_{h,i})^{-1}\,{}^{b}T_{h,j}\) (from the robot); the camera moved by \(B_{ij}={}^{c_i}T_{c_j}=P_i\,P_j^{-1}\) (from the camera; the unknown board pose in the base frame cancels in the product). Hand and camera form one rigid body, so
FIG 3 是整个流程：一台 Robot.arm6，腕上有相机（紫色是真实的 \(X\)，琥珀色是算法估计的 \(\hat X\)），目标是固定在前方的棋盘。手臂的每个姿态都由逆运动学让相机"盯着"目标，围绕目标转动。你选三种运动：只绕一根轴、绕两根轴、或者任意。每个姿态的 \({}^{c}T_{t}\) 带有 PnP 噪声 \(\sigma_R\)（度）与 \(\sigma_t\)（毫米）。下面是两步法：先由旋转轴对齐求 \(R_X\)，再用线性最小二乘求 \(\mathbf t_X\)（5 节推导）。
FIG 3 is the whole pipeline: a Robot.arm6 with a camera on its wrist (purple is the true \(X\), amber the algorithm's estimate \(\hat X\)) and a checkerboard fixed in front. Every arm pose is found by inverse kinematics so that the camera keeps looking at the target while orbiting it. You pick one of three motions: about one axis, about two axes, or arbitrary. Each pose's \({}^{c}T_{t}\) carries PnP noise, \(\sigma_R\) (degrees) and \(\sigma_t\) (millimetres). Below is the two-step method: first \(R_X\) by aligning rotation axes, then \(\mathbf t_X\) by linear least squares (derived in Section 5).
- 选"一根轴"，\(N\ge6\)：轴间夹角为 0，旋转误差高达几十甚至上百度，而且无论加多少个姿态都不会变好。把运动换成"两根轴"，误差立刻降到几度以内。X 中有一个自由度在数据里根本不存在。Choose "one axis" with \(N\ge6\): the axis spread is 0, the rotation error is tens or even over a hundred degrees, and adding poses never helps. Switch the motion to "two axes" and the error falls to within a few degrees at once. One degree of freedom of X simply is not in the data.
- 用"任意"运动，\(N\ge8\)，把噪声调到 \(\sigma_R\le0.3^\circ\)、\(\sigma_t\le2\) mm，幅度 \(\ge25^\circ\)：旋转误差降到 \(1^\circ\) 以下，平移误差降到 8 mm 以下（换几组噪声看看它的波动）。Use the "arbitrary" motion with \(N\ge8\), noise \(\sigma_R\le0.3^\circ\), \(\sigma_t\le2\) mm and amplitude \(\ge25^\circ\): the rotation error falls below \(1^\circ\) and the translation error below 8 mm (try a few noise draws to see how it fluctuates).
试着把"运动幅度"调到 10° 以下再看右图：所有曲线整体上移。小转动使旋转轴 \(\beta_k\)（\(\|\beta_k\|\) 是转角）很短，同样大小的噪声在方向上被放大；此时需要更小的噪声或更多的姿态才能弥补。
Now turn the "motion amplitude" below 10° and look at the right plot: every curve moves up. Small rotations make the axis vectors \(\beta_k\) (whose length is the angle) short, so noise of the same size perturbs their directions much more; you then need less noise or more poses to compensate.
眼在手上，还是眼在手外？Eye-in-hand or eye-to-hand?
另一种常见的装法是eye-to-hand（眼在手外）：相机固定在工作台上，标志物（棋盘或 AprilTag）夹在手爪上。方程的形状完全一样，但每个字母的含义变了：未知的 \(X\) 变成标志物在手里的位姿 \({}^{h}T_{m}\)，\(B\) 是标志物的运动 \(({}^{c}T_{m,i})^{-1}\,{}^{c}T_{m,j}\)（注意顺序和 eye-in-hand 相反）。解出 \(X\) 之后，相机在基座系中的位姿由 \({}^{b}T_{c}={}^{b}T_{h,i}\,X\,({}^{c}T_{m,i})^{-1}\) 给出。还有一种不用相对运动的写法 \(AX=YB\)，把"机器人–世界"和"手–眼"两个未知变换同时解出来。FIG 4 并排画出这三种情形：实线是已知或测得的变换，虚线是要解的未知变换。
The other common mounting is eye-to-hand: the camera is fixed to the workcell and a marker (a checkerboard or an AprilTag) is held in the gripper. The equation keeps exactly the same shape, but every letter changes meaning: the unknown \(X\) becomes the marker's pose in the hand, \({}^{h}T_{m}\), and \(B\) is the marker's motion \(({}^{c}T_{m,i})^{-1}\,{}^{c}T_{m,j}\) (note the order is the opposite of eye-in-hand). Once \(X\) is known, the camera's pose in the base frame follows from \({}^{b}T_{c}={}^{b}T_{h,i}\,X\,({}^{c}T_{m,i})^{-1}\). A third form that avoids relative motions, \(AX=YB\), solves for the "robot–world" and "hand–eye" transforms at the same time. FIG 4 draws the three cases side by side: solid arrows are known or measured transforms, dashed ones are the unknowns to be solved for.
- 依次切换三种配置，各看一遍：每次找出哪条箭头是虚线（未知），并说出 \(A\)、\(B\)、\(X\)（\(Y\)）各对应哪条箭头。Switch through the three configurations and look at each: every time, find the dashed arrows (the unknowns) and say which arrow is \(A\), \(B\), \(X\) (\(Y\)).
刚体运动：同一个运动，在手的坐标系里是 \(A\)，在相机的坐标系里是 \(B\)，\(X\) 是两个坐标系之间的"翻译词典"（共轭 \(B=X^{-1}AX\)）。旋转轴：共轭保持转角，只改变转轴的方向，所以 \(A\) 的转轴 = \(R_X\)（\(B\) 的转轴）；FIG 3 里就是把一堆 \(\beta_k\) 对齐到 \(\alpha_k\)。线性代数：旋转解出后，平移满足 \((R_A-I)\mathbf t_X=R_X\mathbf t_B-\mathbf t_A\)，是一个关于 \(\mathbf t_X\) 的线性方程，而 \(R_A-I\) 有一个沿 \(A\) 转轴的零空间。
Rigid motion: the same motion is \(A\) in the hand's frame and \(B\) in the camera's, and \(X\) is the "dictionary" between the two frames (a conjugation, \(B=X^{-1}AX\)). Rotation axes: conjugation preserves the angle and only changes the axis, so the axis of \(A\) equals \(R_X\) times the axis of \(B\); in FIG 3 this is aligning the \(\beta_k\) to the \(\alpha_k\). Linear algebra: once the rotation is known, the translation obeys \((R_A-I)\mathbf t_X=R_X\mathbf t_B-\mathbf t_A\), linear in \(\mathbf t_X\), and \(R_A-I\) has a null space along the axis of \(A\).
5再看数学The math
提示：把鼠标悬停在公式中带底色的符号上，FIG 1 或 FIG 3 中对应的元素会被高亮。Tip: hover the highlighted symbols in the formulas; the matching element in FIG 1 or FIG 3 lights up.
Zhang 方法：从单应到 \(K\)Zhang's method: from homographies to \(K\)
棋盘上的点 \((X,Y,0)\)（板坐标系，\(Z=0\)）按 10.3 节的投影 \(\lambda\tilde{\mathbf x}=K[R\,|\,\mathbf t]\tilde{\mathbf X}\) 成像，这里 \([R\,|\,\mathbf t]\) 是外参 \({}^{c}T_{b}\)，\(R=[\mathbf r_1\ \mathbf r_2\ \mathbf r_3]\)。因为 \(Z=0\)，\(\mathbf r_3\) 这一列被乘零，剩下：
A board point \((X,Y,0)\) (board frame, \(Z=0\)) is imaged as in Lesson 10.3, \(\lambda\tilde{\mathbf x}=K[R\,|\,\mathbf t]\tilde{\mathbf X}\), where \([R\,|\,\mathbf t]\) is the extrinsic \({}^{c}T_{b}\) and \(R=[\mathbf r_1\ \mathbf r_2\ \mathbf r_3]\). Because \(Z=0\), the column \(\mathbf r_3\) is multiplied by zero, leaving:
其中 \(\mu\) 是单应未知的整体尺度（DLT 只能确定到尺度）。\(H\) 由 54 对点用 DLT 得到（10.5 节）。把 \(\mathbf r_i=\mu^{-1}K^{-1}\mathbf h_i\) 代入旋转矩阵两列满足的 \(\mathbf r_1^\top\mathbf r_2=0\) 与 \(\|\mathbf r_1\|=\|\mathbf r_2\|\)，尺度 \(\mu\) 自己消掉：
where \(\mu\) is the unknown overall scale of the homography (the DLT fixes it only up to scale). \(H\) comes from the 54 point pairs by DLT (Lesson 10.5). Substituting \(\mathbf r_i=\mu^{-1}K^{-1}\mathbf h_i\) into the two conditions on the columns of a rotation, \(\mathbf r_1^\top\mathbf r_2=0\) and \(\|\mathbf r_1\|=\|\mathbf r_2\|\), makes the scale \(\mu\) drop out:
\(B\) 对称，只有 6 个数 \(\mathbf b=(B_{11},B_{12},B_{22},B_{13},B_{23},B_{33})^\top\)，而 \(\mathbf h_i^\top B\,\mathbf h_j=\mathbf v_{ij}^\top\mathbf b\) 对 \(\mathbf b\) 是线性的，其中 \(\mathbf v_{ij}=(h_{1i}h_{1j},\ h_{1i}h_{2j}+h_{2i}h_{1j},\ h_{2i}h_{2j},\ h_{3i}h_{1j}+h_{1i}h_{3j},\ h_{3i}h_{2j}+h_{2i}h_{3j},\ h_{3i}h_{3j})^\top\)。\(n\) 张图叠成 \(2n\times6\) 矩阵：
\(B\) is symmetric with six numbers \(\mathbf b=(B_{11},B_{12},B_{22},B_{13},B_{23},B_{33})^\top\), and \(\mathbf h_i^\top B\,\mathbf h_j=\mathbf v_{ij}^\top\mathbf b\) is linear in \(\mathbf b\), with \(\mathbf v_{ij}=(h_{1i}h_{1j},\ h_{1i}h_{2j}+h_{2i}h_{1j},\ h_{2i}h_{2j},\ h_{3i}h_{1j}+h_{1i}h_{3j},\ h_{3i}h_{2j}+h_{2i}h_{3j},\ h_{3i}h_{3j})^\top\). Stack \(n\) images into a \(2n\times6\) matrix:
然后 \(K\) 由 \(B=K^{-\top}K^{-1}\) 的 Cholesky 分解（或 Zhang 的显式公式）得到：
Then \(K\) follows from the Cholesky factorization of \(B=K^{-\top}K^{-1}\) (or Zhang's explicit formulas):
（\(\mathbf b\) 只确定到符号和尺度；取 \(B_{11}>0\) 的那个符号。主点 \((u_0,v_0)=(c_x,c_y)\)。）有了 \(K\)，每张图的外参是 \(\mathbf r_1=\lambda K^{-1}\mathbf h_1\)、\(\mathbf r_2=\lambda K^{-1}\mathbf h_2\)、\(\mathbf r_3=\mathbf r_1\times\mathbf r_2\)、\(\mathbf t=\lambda K^{-1}\mathbf h_3\)，\(\lambda=1/\|K^{-1}\mathbf h_1\|\)，并取 \(t_z>0\)（板在相机前面）。噪声使得 \([\mathbf r_1\,\mathbf r_2\,\mathbf r_3]\) 不严格是旋转矩阵，用 SVD 取最近的旋转 \(UV^\top\)（1.12 节）。
(\(\mathbf b\) is determined only up to sign and scale; take the sign with \(B_{11}>0\). The principal point is \((u_0,v_0)=(c_x,c_y)\).) With \(K\) in hand, each image's extrinsics are \(\mathbf r_1=\lambda K^{-1}\mathbf h_1\), \(\mathbf r_2=\lambda K^{-1}\mathbf h_2\), \(\mathbf r_3=\mathbf r_1\times\mathbf r_2\), \(\mathbf t=\lambda K^{-1}\mathbf h_3\), \(\lambda=1/\|K^{-1}\mathbf h_1\|\), choosing the sign with \(t_z>0\) (the board is in front of the camera). Noise makes \([\mathbf r_1\,\mathbf r_2\,\mathbf r_3]\) not exactly a rotation, so take the nearest rotation \(UV^\top\) by SVD (Lesson 1.12).
数方程：为什么 3 张，为什么要不同朝向Counting equations: why 3 views, and why different orientations
\(\mathbf b\) 有 6 个分量但只确定到尺度，所以自由度是 5，恰好对应 \((f_x,f_y,s,c_x,c_y)\) 五个内参。每张图 2 个方程，\(n\) 张图 \(2n\) 个：\(2n\ge5\Rightarrow n\ge3\)。若假定 \(s=0\)（现代传感器几乎总成立，OpenCV 默认如此），\(B_{12}=0\)，只剩 4 个自由度，\(n\ge2\) 即可。再假定 \(f_x=f_y\)，且主点在图像中心，一张图就够求焦距。条件是方程独立：若所有棋盘与传感器平行（\(R=R_z\)），\(\mathbf h_1,\mathbf h_2\) 的第三个分量都是 0，\(V\) 的每一行只含 \(B_{11},B_{12},B_{22}\)，\(B_{13},B_{23},B_{33}\) 从来没有出现，主点与焦距不可恢复，FIG 1 里"全部正对相机"就是这样失败的。所有棋盘都绕同一根轴倾斜（FIG 1 的"单轴倾斜"）也是一种奇异配置：此时有一族近似等价的解，焦距会差出几百像素而主点仍然看似正常。这类退化配置的系统分析见 Sturm 与 Maybank (1999)。
\(\mathbf b\) has six components but is fixed only up to scale, hence 5 degrees of freedom, exactly the five intrinsics \((f_x,f_y,s,c_x,c_y)\). Each image gives two equations, \(n\) images give \(2n\): \(2n\ge5\Rightarrow n\ge3\). If you assume \(s=0\) (almost always true for modern sensors, and OpenCV's default), then \(B_{12}=0\) and only 4 degrees of freedom remain, so \(n\ge2\) suffices. Assume also \(f_x=f_y\) with the principal point at the image centre and one image can give the focal length. The condition is that the equations be independent: if every board is parallel to the sensor (\(R=R_z\)), the third components of \(\mathbf h_1,\mathbf h_2\) are 0, every row of \(V\) contains only \(B_{11},B_{12},B_{22}\), the entries \(B_{13},B_{23},B_{33}\) never appear, and principal point and focal length cannot be recovered; this is exactly how "all parallel" fails in FIG 1. Boards that are all tilted about the same axis (FIG 1's "one tilt axis") are also a singular configuration: a family of nearly equivalent solutions appears, the focal length can be off by hundreds of pixels while the principal point still looks normal. A systematic analysis of these degenerate configurations is in Sturm and Maybank (1999).
重投影误差就是最大似然Reprojection error is maximum likelihood
设检测到的角点 \(\mathbf x_{ij}=\pi(K,\mathbf k;\,R_i\mathbf X_j+\mathbf t_i)+\boldsymbol\varepsilon_{ij}\)，\(\boldsymbol\varepsilon_{ij}\sim\mathcal N(\mathbf 0,\sigma^2I_2)\) 相互独立（\(\pi\) 是 10.3 节的带畸变投影）。负对数似然为 \(\frac1{2\sigma^2}\sum\|\boldsymbol\varepsilon_{ij}\|^2\) 加常数，所以最大似然就是
Let the detected corners be \(\mathbf x_{ij}=\pi(K,\mathbf k;\,R_i\mathbf X_j+\mathbf t_i)+\boldsymbol\varepsilon_{ij}\) with independent \(\boldsymbol\varepsilon_{ij}\sim\mathcal N(\mathbf 0,\sigma^2I_2)\) (\(\pi\) is Lesson 10.3's projection with distortion). The negative log-likelihood is \(\frac1{2\sigma^2}\sum\|\boldsymbol\varepsilon_{ij}\|^2\) plus a constant, so maximum likelihood is
LM 的一步解 \((J^\top J+\lambda\,\mathrm{diag}(J^\top J))\,\delta=-J^\top\mathbf r\)（6.6、6.9 节）；\(\lambda\) 大时像梯度下降（稳），小时像 Gauss–Newton（快），按"是否真的下降"自动调节。旋转用旋转向量参数化，更新是对旋转向量的加法，在小转角范围内没有问题（更严格的做法是 6.10 节在流形上更新）。
One LM step solves \((J^\top J+\lambda\,\mathrm{diag}(J^\top J))\,\delta=-J^\top\mathbf r\) (Lessons 6.6 and 6.9); large \(\lambda\) behaves like gradient descent (safe), small \(\lambda\) like Gauss–Newton (fast), adapted by whether the cost really decreased. Rotations are parameterized as rotation vectors and updated additively, which is fine for moderate angles (the stricter way is to update on the manifold, Lesson 6.10).
两个检验值得牢记。噪声下限：模型正确时，残差平方和的期望是 \((N-p)\sigma^2\)，所以每个坐标上的 RMS 约为 \(\sigma\sqrt{1-p/N}\)（FIG 2 的虚线，二维距离的 RMS 再乘 \(\sqrt2\)）。参数不确定度：\(\mathrm{Cov}(\hat{\mathbf p})\approx\hat\sigma^2(J^\top J)^{-1}\)，\(\hat\sigma^2=\|\mathbf r\|^2/(N-p)\)（4.5 节的不确定性传播，也是 Cramér–Rao 下界）。RMS 只回答"拟合得多贴"，协方差才回答"参数有多可信"：参数之间可以高度相关（\(f\) 与到板的距离、\(f\) 与 \(k_1\)），RMS 对沿相关方向的偏移几乎不敏感，这就是 FIG 2 里"RMS 很好但 \(f\) 的标准差很大"的原因。
Two checks are worth memorizing. The noise floor: for a correct model the expected sum of squared residuals is \((N-p)\sigma^2\), so the RMS per coordinate is about \(\sigma\sqrt{1-p/N}\) (the dashed line in FIG 2; the RMS of the 2-D distance is \(\sqrt2\) times that). Parameter uncertainty: \(\mathrm{Cov}(\hat{\mathbf p})\approx\hat\sigma^2(J^\top J)^{-1}\) with \(\hat\sigma^2=\|\mathbf r\|^2/(N-p)\) (the uncertainty propagation of Lesson 4.5, and the Cramér–Rao bound). The RMS answers "how tightly does it fit", only the covariance answers "how far can each parameter be trusted": parameters can be strongly correlated (\(f\) with the distance to the board, \(f\) with \(k_1\)), and the RMS is nearly blind to a shift along a correlated direction, which is why FIG 2 can show a fine RMS together with a large standard deviation of \(f\).
手眼标定：推导Hand–eye calibration: the derivation
| 符号Symbol | 含义Meaning | 来源Source |
|---|---|---|
| \({}^{b}T_{h,i}\) | 姿态 \(i\) 的手（工具）在基座系的位姿hand (tool) pose in the base frame at pose \(i\) | 正运动学 \(\mathrm{FK}(\mathbf q_i)\)（7.3 节）forward kinematics \(\mathrm{FK}(\mathbf q_i)\) (7.3) |
| \(P_i={}^{c}T_{t,i}\) | 目标在相机系的位姿（"相机←目标"）target pose in the camera frame ("camera ← target") | PnP 测量（11.4 节），有噪声PnP measurement (11.4), noisy |
| \(A_{ij}\) | \(({}^{b}T_{h,i})^{-1}\,{}^{b}T_{h,j}\) | 手在其自身坐标系中的相对运动hand motion expressed in the hand frame |
| \(B_{ij}\) | \(P_i\,P_j^{-1}\) | 相机在其自身坐标系中的相对运动camera motion expressed in the camera frame |
| \(X\) | \({}^{h}T_{c}=(R_X,\mathbf t_X)\) | 未知：相机在手里的位姿unknown: the camera's pose in the hand |
推导：相机的位姿有两种写法，\({}^{b}T_{c,i}={}^{b}T_{h,i}X\)，也是 \({}^{b}T_{t}\,P_i^{-1}\)。于是 \(B_{ij}=({}^{b}T_{c,i})^{-1}\,{}^{b}T_{c,j}=P_i\,{}^{b}T_{t}^{-1}\,{}^{b}T_{t}\,P_j^{-1}=P_iP_j^{-1}\)，同时 \(B_{ij}=X^{-1}({}^{b}T_{h,i})^{-1}\,{}^{b}T_{h,j}X=X^{-1}A_{ij}X\)。按分块 \((R,\mathbf t)\) 展开 \(AX=XB\)：
Derivation: the camera pose can be written two ways, \({}^{b}T_{c,i}={}^{b}T_{h,i}X\) and also \({}^{b}T_{t}\,P_i^{-1}\). Hence \(B_{ij}=({}^{b}T_{c,i})^{-1}\,{}^{b}T_{c,j}=P_i\,{}^{b}T_{t}^{-1}\,{}^{b}T_{t}\,P_j^{-1}=P_iP_j^{-1}\), and also \(B_{ij}=X^{-1}({}^{b}T_{h,i})^{-1}\,{}^{b}T_{h,j}X=X^{-1}A_{ij}X\). Expanding \(AX=XB\) in blocks \((R,\mathbf t)\):
旋转部分：\(R_A=R_XR_BR_X^\top\) 是共轭，转角相同，转轴满足 \(\boldsymbol\alpha=R_X\boldsymbol\beta\)，其中 \(\boldsymbol\alpha=\mathrm{Log}(R_A)\)，\(\boldsymbol\beta=\mathrm{Log}(R_B)\)（5.6 节的旋转向量）。一个运动只给出一个向量对，\(R_X\) 还剩一个绕 \(\boldsymbol\beta\) 的自由度；两个转轴不平行的运动给出两个不共线的向量对，\(R_X\) 就唯一了。有 \(m\) 个运动（含噪声）时，最小二乘是 Kabsch 问题（不去中心化，因为 \(\boldsymbol\alpha=R_X\boldsymbol\beta\) 里没有平移）：
Rotation part: \(R_A=R_XR_BR_X^\top\) is a conjugation, so the angles agree and the axes satisfy \(\boldsymbol\alpha=R_X\boldsymbol\beta\), where \(\boldsymbol\alpha=\mathrm{Log}(R_A)\) and \(\boldsymbol\beta=\mathrm{Log}(R_B)\) (the rotation vectors of Lesson 5.6). One motion gives one vector pair and leaves a rotation about \(\boldsymbol\beta\) free in \(R_X\); two motions with non-parallel axes give two non-collinear pairs and \(R_X\) becomes unique. With \(m\) noisy motions, the least-squares problem is a Kabsch problem (without centring, because \(\boldsymbol\alpha=R_X\boldsymbol\beta\) has no translation):
这与 Park 和 Martin (1994) 在欧氏群上的闭式解等价。平移部分：代入 \(\hat R_X\)，每个运动给 3 个关于 \(\mathbf t_X\) 的线性方程，叠起来是 \(3m\times3\) 的最小二乘（1.13 节）。矩阵 \(R_A-I\) 的零空间是 \(A\) 的转轴方向（\((R_A-I)\mathbf a=\mathbf 0\)），所以每个运动只约束了垂直于它转轴的 2 个分量；转轴不平行的运动叠在一起才满秩。
This is equivalent to the closed form of Park and Martin (1994) on the Euclidean group. Translation part: substituting \(\hat R_X\), each motion gives three linear equations in \(\mathbf t_X\), stacked into a \(3m\times3\) least-squares problem (Lesson 1.13). The null space of \(R_A-I\) is the direction of the axis of \(A\) (\((R_A-I)\mathbf a=\mathbf 0\)), so each motion constrains only the 2 components perpendicular to its axis; motions with non-parallel axes stacked together make the matrix full rank.
不可观的方向The unobservable direction
当所有运动都是绕同一条直线 \(L\) 的转动（FIG 3 的"一根轴"）时，设 \(S\) 是相机系里绕 \(L\) 转 \(\varphi\)（并可沿 \(L\) 平移 \(d\)）的螺旋运动。\(S\) 与每一个 \(B_k\) 可交换，于是 \(A_k(XS)=X B_kS=(XS)B_k\)：\(X\) 和 \(XS\) 对全部数据同样成立，一个 2 参数的解族。这不是算法不够好，而是数据中没有信息：相机绕那条轴转一个角，或者沿它滑一段，全部测量一模一样（棋盘的未知位姿吸收了差别）。数值上它表现为 \(\sum\boldsymbol\beta_k\boldsymbol\alpha_k^\top\) 的秩为 1（FIG 3 读数的 \(\sigma_2/\sigma_1\approx0\)），以及 \(R_A-I\) 的叠加矩阵秩为 2。
When all motions are rotations about one line \(L\) (FIG 3's "one axis"), let \(S\) be the screw motion in the camera frame that rotates by \(\varphi\) about \(L\) (and may slide by \(d\) along it). \(S\) commutes with every \(B_k\), so \(A_k(XS)=XB_kS=(XS)B_k\): \(X\) and \(XS\) fit all the data equally well, a two-parameter family of solutions. This is not a weakness of the algorithm but missing information: rotating the camera about that axis, or sliding it along, leaves every measurement identical (the board's unknown pose absorbs the difference). Numerically it shows up as \(\sum\boldsymbol\beta_k\boldsymbol\alpha_k^\top\) having rank 1 (FIG 3's \(\sigma_2/\sigma_1\approx0\)) and the stacked \(R_A-I\) having rank 2.
误差度量。标定好 \(\hat X\) 后，旋转误差用 \(\|\mathrm{Log}(\hat R_X^\top R_X)\|\)（度），平移误差用 \(\|\hat{\mathbf t}_X-\mathbf t_X\|\)（毫米）；真实标定时没有真值，只能看一致性：同一个固定目标，在手臂不同姿态下经 \({}^{b}T_{h,i}\hat X P_i\) 算出的基座系位置应当重合（FIG 5）。
Error metrics. After calibration, measure the rotation error as \(\|\mathrm{Log}(\hat R_X^\top R_X)\|\) (degrees) and the translation error as \(\|\hat{\mathbf t}_X-\mathbf t_X\|\) (millimetres). In a real calibration there is no ground truth and only consistency remains: a fixed target, computed through \({}^{b}T_{h,i}\hat X P_i\) from different arm poses, must land at the same base-frame position (FIG 5).
两步法的局限，与其他解法。上面的两步法先解旋转再解平移，旋转的误差会传到平移；而且它最小化的是运动方程的残差，不是图像里的误差。更好的做法是把它当作初值，再对图像重投影误差同时优化 \(X\) 与 \({}^{b}T_{t}\)（和 FIG 2 一样的 LM）。其他经典解法：Tsai 与 Lenz (1989)（先解旋转，再解平移的早期线性法），Daniilidis (1999)（对偶四元数，同时解旋转与平移），以及李群/Kronecker 积形式的 \(AX=YB\) 求解器（Shah, 2013）。
Limits of the two-step method, and other solvers. The two-step method above solves rotation first and translation second, so rotation error leaks into translation; and it minimizes the residual of the motion equation, not the error in the image. A better recipe uses it as the initial guess and then optimizes \(X\) and \({}^{b}T_{t}\) jointly on the image reprojection error (the same LM as in FIG 2). Other classics: Tsai and Lenz (1989), an early method that solves rotation then translation linearly; Daniilidis (1999), dual quaternions that solve rotation and translation together; and Lie-group or Kronecker-product solvers for \(AX=YB\) (Shah, 2013).
6机器人中的它：相机引导的抓取到底准不准In robotics: is the camera-guided pick actually accurate?
标定做得好不好，最终看机器人在做什么。典型流程：相机在图像里找到目标，PnP 或深度图给出目标在相机系里的位姿 \(P\)，机器人用 \({}^{b}T_{h}\,\hat X\,P\) 把它搬到基座系，再规划去抓。如果 \(\hat X\) 有误差，抓取点就会随着手臂姿态而系统性地飘。
FIG 5 用 FIG 3 当前的估计 \(\hat X\) 做这件事。手臂访问 20 个新的随机姿态（与标定用的姿态无关，噪声也是新的），每个姿态下相机测到同一个固定目标，我们把它换算成基座系位置。如果标定与测量都完美，所有预测会落在同一个点；散开的云就是标定重复性。青色用真实的 \(X\)（只剩传感器噪声的底线），琥珀色用估计的 \(\hat X\)。琥珀云明显大于青云，或者中心偏离十字，说明标定误差在主导。回到 FIG 3 改善标定（更多姿态、更丰富的运动、更小的噪声），看琥珀云收缩到青云。
A calibration is judged by what the robot does. The typical pipeline: the camera finds the target in the image, PnP or a depth map gives the target's pose \(P\) in the camera frame, the robot carries it to the base frame with \({}^{b}T_{h}\,\hat X\,P\), and then plans the grasp. If \(\hat X\) is off, the grasp point drifts systematically with the arm pose.
FIG 5 does this with FIG 3's current estimate \(\hat X\). The arm visits 20 new random poses (independent of the calibration poses, with fresh noise); at each the camera measures the same fixed target and we convert it to a base-frame position. With a perfect calibration and perfect measurements all predictions would land on one point; the scattered cloud is the calibration repeatability. Cyan uses the true \(X\) (the floor set by sensor noise alone), amber uses the estimate \(\hat X\). If the amber cloud is clearly larger than the cyan one, or its centre is off the cross, calibration error dominates. Go back to FIG 3 and improve the calibration (more poses, richer motion, less noise) to watch the amber cloud shrink onto the cyan one.
- 回到 FIG 3 选"一根轴"，看琥珀云散开到青色底线的 3 倍以上；再改用"任意"运动、足够大的幅度和足够多的姿态，让琥珀云的散布回到青色底线的 1.5 倍以内。Go back to FIG 3, choose "one axis", and watch the amber cloud spread to over 3 times the cyan floor; then switch to the "arbitrary" motion with a large amplitude and enough poses until the amber spread is within 1.5 times the cyan floor.
内参标定：OpenCV 的 findChessboardCorners + cornerSubPix + calibrateCamera 就是本节的流程（Zhang 初值 + LM 精化，支持径向与切向畸变，另有鱼眼模型 cv::fisheye）；ROS 的 camera_calibration 包在它外面套了一层交互界面。标定板：纯棋盘格要求整块板都可见，ChArUco（棋盘 + ArUco 标记）和 AprilTag 网格允许局部可见、自带编号，对遮挡和大角度更稳健。多传感器：Kalibr（ETH）标定多相机、相机–IMU，并估计时间偏移（Furgale 等，IROS 2013）。手眼：OpenCV 有 calibrateHandEye（Tsai、Park、Horaud、Andreff、Daniilidis 几种方法）与 calibrateRobotWorldHandEye（\(AX=YB\)）；ROS 里的 MoveIt 手眼标定插件与 easy_handeye 把采集姿态、检测标记、求解封装成了工作流。出厂与用户标定：RealSense、ZED 这类相机出厂时已逐台标定并把参数存在设备里，但温度漂移、镜头受力、更换光圈都可能让它偏离，高精度应用仍要现场重标。学习策略：以点云、体素或多视角相机为输入的 3D 感知策略（例如 PerAct 一类）把像素用相机内外参放进统一的三维空间，外参错 1 厘米，动作目标就错 1 厘米，所以在数据采集前必须先标定并记录。运动学标定是同一类问题的第三种：连杆长度、关节零位的小误差也会让 \(\mathrm{FK}(\mathbf q)\) 偏差，可用 7.3 节的正运动学对运动学参数求导（7.4 节的雅可比的"参数版"），同样用最小二乘估计；手眼标定的结果里不可避免地混进了这类误差。
Intrinsics: OpenCV's findChessboardCorners + cornerSubPix + calibrateCamera is exactly this lesson's pipeline (Zhang start plus LM refinement, radial and tangential distortion, and a separate fisheye model cv::fisheye); ROS's camera_calibration package wraps an interactive GUI around it. Targets: a plain checkerboard needs the whole board visible; ChArUco (checkerboard + ArUco markers) and AprilTag grids work with partial views and carry their own IDs, so they are sturdier against occlusion and extreme angles. Multi-sensor: Kalibr (ETH) calibrates multi-camera rigs and camera–IMU pairs including the time offset (Furgale et al., IROS 2013). Hand–eye: OpenCV has calibrateHandEye (the Tsai, Park, Horaud, Andreff and Daniilidis methods) and calibrateRobotWorldHandEye (\(AX=YB\)); MoveIt's hand–eye calibration plugin and easy_handeye in ROS package the workflow of collecting poses, detecting the marker and solving. Factory versus user calibration: cameras such as RealSense and ZED are calibrated per unit at the factory and store the parameters on the device, but temperature drift, lens stress or a changed aperture can move them away, so high-accuracy applications still recalibrate on site. Learned policies: 3D-aware policies that take point clouds, voxels or multiple camera views (PerAct-style, for example) place pixels in one 3-D space using the camera intrinsics and extrinsics; an extrinsic error of 1 cm becomes a 1 cm error in the action target, so calibrate before collecting data and record the calibration with it. Kinematic calibration is a third problem of the same kind: small errors in link lengths and joint zero offsets also bend \(\mathrm{FK}(\mathbf q)\), and can be estimated by least squares from derivatives of forward kinematics (Lesson 7.3) with respect to the kinematic parameters (a "parameter version" of Lesson 7.4's Jacobian); the hand–eye result inevitably absorbs some of those errors.
7常见误区Common pitfalls
20 张几乎正对相机、只是平移了一下的图，信息量接近 0（FIG 1 的"全部正对相机"）；20 张都绕同一根轴倾斜，也只约束了一半。好的标定集要覆盖多种倾斜方向、图像的四角（畸变在边缘最明显，中央的角点对 \(k_1\) 几乎没有约束）、以及几种距离。手眼同理：FIG 3 里"一根轴"的运动，加到 100 个姿态也不会变好，因为缺的不是样本数而是方向。
Twenty images that all face the camera and were merely translated carry almost no information (FIG 1's "all parallel"); twenty tilted about one common axis constrain only half. A good set covers several tilt directions, the corners of the image (distortion is strongest at the edge and central corners barely constrain \(k_1\)), and several distances. The same holds for hand–eye: FIG 3's "one axis" motion does not improve with 100 poses, because what is missing is not samples but directions.
方格尺寸输错，\(K\) 不变而所有平移（和手眼的 \(\mathbf t_X\)）按同一比例错，重投影误差一点也不会报警（FIG 1 的方格尺寸滑块）。常见来源是打印机"适应页面"悄悄缩放了标定板；用卡尺量几个方格才可靠。板子要平：贴在硬板上，不要用软纸或会翘的亚克力。运动模糊、卷帘快门会让角点位置偏移，所以采集时让机器人停稳再拍；机器人关节角和图像的时间戳要对得上，否则 \({}^{b}T_{h}\) 与 \(P\) 来自不同的时刻，手眼误差就是这一段位移。
If the square size is entered wrong, \(K\) is unaffected but all translations (and the hand–eye \(\mathbf t_X\)) are wrong by the same ratio and the reprojection error does not alarm at all (FIG 1's square-size slider). A common source is a printer's "fit to page" silently scaling the target; measure several squares with calipers. The board must be flat: mount it on a rigid backing, not soft paper or warping acrylic. Motion blur and rolling shutter shift the corners, so stop the robot before shooting; joint angles and image timestamps must match, otherwise \({}^{b}T_{h}\) and \(P\) come from different instants and the hand–eye error is that displacement.
焦距随对焦和变焦变化（自动对焦的"呼吸效应"可以让 \(f\) 变化百分之几），所以标定时要锁定对焦与光圈，使用时也一样。分辨率变了，\(K\) 也要跟着缩放：缩放因子 \(s\) 时 \(f\to sf\)，而主点在"像素中心为整数坐标"的约定下是 \(c\to s(c+0.5)-0.5\)，忘了 0.5 会引入半个像素的偏差。裁剪（crop）则只平移主点。相机在支架上被碰过、热了之后，外参也不再是标定时的值，最好定期用 FIG 5 的重复性检验来发现它。
Focal length changes with focus and zoom (autofocus "breathing" can shift \(f\) by a few percent), so lock focus and aperture during calibration and use. If the resolution changes, \(K\) must be rescaled: with scale factor \(s\), \(f\to sf\), while the principal point under the "pixel centres at integer coordinates" convention becomes \(c\to s(c+0.5)-0.5\); forgetting the 0.5 introduces half a pixel of error. Cropping only shifts the principal point. A camera that was bumped or warmed up no longer has its calibration-day extrinsics either; check periodically with FIG 5's repeatability test to catch it.
教科书、论文和库在这里各不相同：\(A\) 用手的运动还是基座系里的运动，\(B\) 用 \(P_iP_j^{-1}\) 还是 \(P_i^{-1}P_j\)，\(P\) 是"目标在相机里"还是"相机在目标里"，eye-in-hand 与 eye-to-hand 的 \(X\) 又分别是什么。只要有一处求逆搞反，算法照样输出一个看起来像样的旋转矩阵和一个"合理"的平移，只是错的。防线有两条：一是写下每个变换的 \({}^{A}T_{B}\) 记号并检查下标能否像链条一样首尾相接；二是永远做 FIG 5 的一致性检验，用没参与标定的姿态预测固定目标的位置。
Textbooks, papers and libraries differ here: \(A\) as the hand's own motion or as the motion in the base frame, \(B\) as \(P_iP_j^{-1}\) or \(P_i^{-1}P_j\), \(P\) as "target in camera" or "camera in target", and what \(X\) is in eye-in-hand versus eye-to-hand. A single inverted transform still makes the algorithm return a plausible-looking rotation matrix and a "reasonable" translation; they are simply wrong. There are two defences: write every transform in the \({}^{A}T_{B}\) notation and check that the indices chain head to tail; and always run FIG 5's consistency check, predicting the fixed target's position from poses that took no part in the calibration.
FIG 2 里 3 张图就能把 RMS 压到噪声下限，但 \(f\) 的标准差有十几个像素：参数相关（\(f\) 与距离、\(f\) 与 \(k_1\)）让误差沿一个 RMS 几乎看不见的方向漂。看标准差、看相关系数、留一部分图当验证集，比只看 RMS 可靠。畸变模型也不是越多越好：用 \(k_3\)、切向项去拟合边缘很少被覆盖的区域，会产生在图像外面乱飞的"畸变曲线"。手眼标定则被小转动卡死：转角小，\(\boldsymbol\beta_k\) 很短，噪声使其方向严重失真；经验上，让相对转动的角度至少有十几度到几十度，并且转轴方向丰富（FIG 3 的"运动幅度"）。
In FIG 2 three images already push the RMS down to the noise floor, yet \(f\) has a standard deviation of a dozen pixels: parameter correlations (\(f\) with distance, \(f\) with \(k_1\)) let the error drift along a direction the RMS can hardly see. Looking at standard deviations, at correlations, and holding out some images as a validation set is more reliable than the RMS alone. More distortion terms are not always better either: fitting \(k_3\) and tangential terms to regions the data rarely cover produces a distortion curve that flies off outside the covered area. Hand–eye is held hostage by small rotations: a small angle makes \(\boldsymbol\beta_k\) short, so noise ruins its direction; as a rule of thumb the relative rotations should span at least a dozen to a few dozen degrees, with axes of varied directions (FIG 3's "motion amplitude").
8小练习Check yourself
9延伸阅读Further reading
- Z. Zhang, A Flexible New Technique for Camera Calibration, IEEE TPAMI 22(11), 2000 — 本节第一部分的原始论文：单应、约束、闭式解与最大似然精化The source of the first half of this lesson: homographies, constraints, closed form and maximum-likelihood refinement
- R. Hartley & A. Zisserman, Multiple View Geometry in Computer Vision (2nd ed., Cambridge University Press, 2004), Ch. 7–8 — 相机矩阵的估计、相机标定与绝对二次曲线的几何Estimating the camera matrix, calibration, and the geometry of the absolute conic
- P. Sturm & S. Maybank, On Plane-Based Camera Calibration: A General Algorithm, Singularities, Applications, CVPR 1999 — 平面标定的奇异（退化）配置The singular (degenerate) configurations of plane-based calibration
- R. Y. Tsai & R. K. Lenz, A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration, IEEE Trans. Robotics and Automation 5(3), 1989 — 手眼标定的经典早期方法An early classic of hand–eye calibration
- F. C. Park & B. J. Martin, Robot Sensor Calibration: Solving AX=XB on the Euclidean Group, IEEE Trans. Robotics and Automation 10(5), 1994 — 用李群的对数求解 \(AX=XB\)，本节的旋转步骤Solving \(AX=XB\) with the group logarithm; the rotation step of this lesson
- K. Daniilidis, Hand-Eye Calibration Using Dual Quaternions, Int. J. Robotics Research 18(3), 1999 — 用对偶四元数同时求旋转与平移Solving rotation and translation simultaneously with dual quaternions
- M. Shah, Solving the Robot-World/Hand-Eye Calibration Problem Using the Kronecker Product, J. Mechanisms and Robotics 5(3), 2013 — \(AX=YB\) 的求解Solving \(AX=YB\)
- P. Furgale, J. Rehder & R. Siegwart, Unified Temporal and Spatial Calibration for Multi-Sensor Systems, IROS 2013 — Kalibr：多相机、相机–IMU 与时间偏移的标定Kalibr: calibrating multi-camera rigs, camera–IMU pairs and time offsets
- OpenCV, Camera Calibration and 3D Reconstruction (calib3d) documentation — 本节流程的工程实现An engineering implementation of this lesson's pipeline