针孔相机模型The Pinhole Camera

相机便宜、轻、信息量巨大：一帧图像就有几十万到几百万个像素，而且它几乎是机器人唯一能"看见"颜色、纹理、文字和语义的传感器。抓取要知道杯子在哪里，导航要知道门在哪里，视觉伺服要把手爪对准螺丝孔，模仿学习的策略网络直接以图像为输入。但图像里没有一个数字写着"这个点离你 2.3 米"。要把像素变回三维世界，第一件事就是弄清楚：相机到底丢掉了什么，保留了什么。

这是 Part III（3D 视觉与几何）的第一节，所以先说一下这一部分在讲什么。它只有一条主线：三维点 → 像素这个映射，以及怎样求它的逆。第 10 章（相机与投影）研究这个映射本身：针孔模型（本节）、齐次坐标（10.2）、内参外参与畸变（10.3）、标定（10.4）、平面到平面的单应（10.5）。第 11 章（多视图几何）问：有了两张以上的图像，怎样恢复深度和相机运动？对极几何、立体视觉、三角化、PnP、特征匹配与光束法平差。第 12 章（三维表示与配准）问：恢复出来的三维结构怎样存储、对齐、渲染？点云与 ICP、体素/网格/SDF、图形学管线、NeRF 与 3D Gaussian Splatting。每一章都是同一个映射的变奏，而它的第一个版本就是这一页。

它和前面的内容有天然联系：像素坐标是一个向量（1.1 节），投影是一个"几乎线性"的映射（1.3 节），它的局部线性化就是雅可比（3.3 节），相机本身是一个坐标系（5.1 节）。装在机械臂末端的相机，位姿就是正运动学（7.3 节）再乘一个固定的手眼变换，没有任何新东西。

Cameras are cheap, light and information-dense: one frame holds hundreds of thousands to millions of pixels, and a camera is nearly the only robot sensor that sees colour, texture, text and semantics. A grasp needs to know where the mug is, navigation where the door is, visual servoing how to line the gripper up with a screw hole, and an imitation-learning policy takes images straight in. Yet no number in an image says "this point is 2.3 m away". To turn pixels back into the 3D world, the first thing to understand is exactly what a camera throws away and what it keeps.

This is the first lesson of Part III (3D Vision & Geometry), so a word on what the part is about. It has a single thread: the map from 3D points to pixels, and how to invert it. Chapter 10 (Cameras & Projection) studies the map itself: the pinhole model (this lesson), homogeneous coordinates (10.2), intrinsics, extrinsics and distortion (10.3), calibration (10.4), plane-to-plane homographies (10.5). Chapter 11 (Multi-View Geometry) asks: given two or more images, how do we recover depth and camera motion? Epipolar geometry, stereo, triangulation, PnP, feature matching and bundle adjustment. Chapter 12 (3D Representations & Registration) asks: how is the recovered structure stored, aligned and rendered? Point clouds and ICP, voxels/meshes/SDFs, the graphics pipeline, NeRF and 3D Gaussian Splatting. Every chapter is a variation on the same map, and its first version is this page.

It connects naturally to what you know: a pixel coordinate is a vector (Lesson 1.1), projection is an "almost linear" map (Lesson 1.3), its local linearization is a Jacobian (Lesson 3.3), and a camera is a coordinate frame (Lesson 5.1). A camera mounted on an arm's wrist has a pose given by forward kinematics (Lesson 7.3) times one fixed hand-eye transform; nothing new there.

最早的"相机"是一间暗室：在墙上开一个小孔，对面的墙上就出现一幅倒立的室外景象。原理只有一条：光沿直线传播。场景中每个点向四面八方发光，其中只有恰好穿过小孔的那一根射线能到达像平面，所以像平面上的每个点只"听到"场景里一个方向的光。

下图是侧视图：针孔在原点，光轴指向右，物体是右边那根琥珀色的箭头（高 \(H\)，距离针孔 \(Z\)），像平面在针孔后面 \(f\) 处（\(f\) 叫焦距，这里指"针孔到像平面的距离"）。过针孔的那条射线把图分成两个相似三角形，一个以物体为边，一个以像为边，所以

The first "camera" was a dark room: cut a small hole in one wall and an upside-down picture of the outdoors appears on the opposite wall. One principle explains it: light travels in straight lines. Every scene point sends light in all directions, but only the one ray that happens to pass through the hole reaches the image plane, so each point of the image plane "hears" light from just one direction of the scene.

The figure is a side view: the pinhole is at the origin, the optical axis points right, the object is the amber arrow on the right (height \(H\), distance \(Z\) from the pinhole), and the image plane sits a distance \(f\) behind the pinhole (\(f\) is the focal length, here simply "pinhole to image plane"). The ray through the pinhole splits the picture into two similar triangles, one with the object as a side and one with the image, so

负号表示像是倒立的。计算机视觉里一般把像平面"搬"到针孔前面同样的距离 \(f\) 处（图中的虚线竖线，叫虚拟像平面）：同一条射线与它相交在同一高度的镜像点，数值相同，像却是正的。以后我们永远用这个正立的约定。拖动箭头尖端，调焦距和孔径，注意三件事：像的大小 \(\propto f/Z\)；孔变大时，一个场景点在像平面上被抹成一个光斑，直径 \(b = a(1+f/Z)\)；而进光量 \(\propto a^2\)。孔也不能太小：太小则太暗，而且光的衍射会让光斑重新变大（最清晰的孔径大致与 \(\sqrt{f\lambda}\) 同量级，\(\lambda\) 是波长）。

The minus sign says the image is upside down. Computer vision usually "moves" the image plane in front of the pinhole at the same distance \(f\) (the dashed vertical line, the virtual image plane): the same ray crosses it at the mirrored point, with the same numbers but an upright picture. We will always use this upright convention from now on. Drag the arrow tip, change the focal length and the aperture, and notice three things: the image size is \(\propto f/Z\); as the hole grows, one scene point is smeared into a blur disc of diameter \(b = a(1+f/Z)\); and the light collected is \(\propto a^2\). The hole cannot be too small either: it gets too dark, and diffraction makes the spot grow again (the sharpest aperture is of the order of \(\sqrt{f\lambda}\), with \(\lambda\) the wavelength).

用透镜升级一下。小孔太暗，所以真实相机用透镜把穿过一个大孔径的整束光重新汇聚到一点。薄透镜公式 \(\tfrac1Z+\tfrac1{z_i}=\tfrac1F\)（\(F\) 是透镜焦距，\(z_i\) 是清晰成像的像距）说明：只有一个深度的平面能恰好聚焦，其他深度会模糊成直径 \(c\approx a\,|s-z_i|/z_i\) 的弥散圆（\(s\) 是传感器到透镜的距离），可接受的模糊范围就是景深；光圈数 \(N=F/a\) 越大（孔径越小）景深越深，越接近针孔。关键是：对"一个点落在哪个像素"这个几何问题，对焦清晰时透镜相机和针孔相机给出同一个答案，唯一的区别是 \(f\) 现在是"透镜到传感器的距离"（对远处物体就是 \(F\) 附近）。所以本课以及整个 Part III 都用针孔模型。

Upgrading to a lens. A pinhole is dim, so real cameras use a lens to re-focus the whole cone of light through a large aperture back to a point. The thin-lens equation \(\tfrac1Z+\tfrac1{z_i}=\tfrac1F\) (\(F\) the lens focal length, \(z_i\) the in-focus image distance) says that only one depth plane is exactly in focus; other depths blur into a circle of diameter \(c\approx a\,|s-z_i|/z_i\) (\(s\) the sensor-to-lens distance), and the range of acceptable blur is the depth of field. A larger f-number \(N=F/a\) (smaller aperture) gives a deeper field, closer to a pinhole. The key point: for the geometric question "which pixel does this point land on", an in-focus lens camera gives the same answer as a pinhole camera; the only difference is that \(f\) is now the lens-to-sensor distance (about \(F\) for distant objects). That is why this lesson, and all of Part III, uses the pinhole model.

现在把相机放进三维世界。左图是世界：地面网格（红线沿 \(x\)，绿线沿 \(y\)）、一个 1 m 的方盒（琥珀色）、三根竖直的杆（蓝色）和一台相机的视锥；相机自带的坐标轴是 OpenCV 光学坐标系：\(x_c\) 向右（红），\(y_c\) 向下（绿），\(z_c\) 向前（蓝）。右图是这台相机拍到的画面，像素坐标 \((u,v)\)：\(u\) 向右，\(v\) 向下，原点在左上角。画面外的淡色线是"如果传感器更大会看到什么"。

用滑块把相机绕场景转、拉远拉近、抬高降低，再调焦距。看三件事：平行线怎样汇聚到消失点（彩色圆圈，与世界轴同色；用同样方式画出的虚线是地平线）；同一个方盒怎样随位置变大变小；以及改变焦距与移动相机有什么不同。

Now put the camera into a 3D world. The left view is the world: a floor grid (red lines along \(x\), green along \(y\)), a 1 m box (amber), three vertical poles (blue) and a camera frustum; the camera's own axes are the OpenCV optical frame: \(x_c\) right (red), \(y_c\) down (green), \(z_c\) forward (blue). The right view is what this camera sees, in pixel coordinates \((u,v)\): \(u\) to the right, \(v\) down, origin at the top-left corner. The faint lines outside the frame show "what a bigger sensor would see".

Use the sliders to orbit the camera around the scene, move it in and out, raise and lower it, and change the focal length. Watch three things: how parallel lines meet at vanishing points (coloured rings, matching the world axes; the dashed line through them is the horizon); how the same box grows and shrinks with position; and how changing the focal length differs from moving the camera.

反过来问：已知一个像素，三维点在哪里？答案是一条射线：沿射线的每一点都投影到这个像素。下图（侧视图，相机在左边，像平面归一化到 \(f=1\)，实际换算用 \(f=800\) px）里，琥珀色手柄在像平面上选像素，拖动射线上的点手柄改变深度，像始终不变。一个近处的小东西，一个远处的大东西，一个更远的巨人，它们的像完全相同，这就是单目的尺度歧义：玩具车模型和真车拍出来分不出来。

怎么解决？再来一台相机。第二台相机的射线与第一条相交于一点，交点就是三维点（深度恢复，11.3 节）。打开"两台相机"，第二台相机里同一个点的像素位置随深度变化，差值就是视差 \(d = f\,b/Z\)（\(b\) 是基线，这里为了画在侧视图里把基线画成竖直的；真实的立体相机是水平的）。

Ask the reverse question: given a pixel, where is the 3D point? The answer is a ray: every point along it projects to that pixel. In the figure below (side view, camera on the left, image plane normalized to \(f=1\), pixel conversion with \(f=800\) px), the amber handle picks the pixel on the image plane and the point handle on the ray changes the depth while the image never changes. A small thing near, a larger thing farther, a giant even farther: their images are identical. This is the scale ambiguity of a single camera: a toy-car model and a real car look the same.

The fix is a second camera. The second camera's ray meets the first at one point, and the intersection is the 3D point (depth recovery, Lesson 11.3). Turn on "two cameras": the pixel of the same point in camera 2 moves with depth, and the difference is the disparity \(d = f\,b/Z\) (\(b\) the baseline, drawn vertically so it fits in this side view; real stereo rigs are horizontal).

很多人以为"广角镜头让透视夸张，长焦镜头把背景压扁"。更准确的说法是：透视的强弱由相机的位置决定，焦距只决定取景范围。把相机从主体前退后的同时拉长焦距，让主体在画面里保持同样大（\(f/Z\) 不变），背景却会"涨"起来：这就是希区柯克《迷魂记》里的"滑动变焦"（dolly zoom，Vertigo effect）。道理很简单：主体深度 \(Z\)、背景深度 \(Z+D\)，同样大小的东西在画面里的比是

Many people believe "wide lenses exaggerate perspective, tele lenses flatten the background". More precisely, perspective strength is set by the camera's position, and the focal length only sets the framing. Back the camera away from the subject while lengthening the focal length so the subject keeps the same size in the frame (\(f/Z\) constant), and the background "swells": this is the dolly zoom (Vertigo effect) from Hitchcock's Vertigo. The reason is simple: with subject depth \(Z\) and background depth \(Z+D\), equal-size things appear in the frame in the ratio

这个比值只与 \(Z\) 和 \(D\) 有关，与 \(f\) 无关。\(Z\to\infty\) 时比值趋于 1（背景与主体一样大，画面"平"，即长焦的压缩感）；\(Z\) 小时背景被甩得很小（广角的"纵深感"）。下图的左半是俯视图（相机在下，光轴向上，青色扇形是视场），右半是相机画面（主体是琥珀色的人，后面是 3 m 和 8 m 处的两排建筑）。选"锁定主体大小"后，拖动距离，焦距会自动跟着变。

This ratio depends only on \(Z\) and \(D\), not on \(f\). As \(Z\to\infty\) it tends to 1 (the background is as big as the subject, a "flat" picture, the compression of a tele lens); for small \(Z\) the background is dwarfed (the "depth" of a wide lens). The left half of the figure below is a top view (camera at the bottom, optical axis up, cyan wedge = field of view) and the right half is the camera image (the amber figure is the subject, with two rows of buildings 3 m and 8 m behind). Choose "lock subject size" and drag the distance: the focal length follows automatically.

当物体的深度起伏 \(\Delta Z\) 远小于它到相机的平均距离 \(Z_0\)，投影公式里的除数 \(Z\) 几乎是常数 \(Z_0\)，投影就近似成一个线性映射：\(u\approx f X/Z_0 + c_x\)，即弱透视（缩放的正交投影）：把三维点丢掉 \(Z\)，整体缩放 \(f/Z_0\)。下图把一个立方体放在深度 \(Z_0\)，琥珀色是真实的透视投影，紫色虚线是弱透视（两者都用 \(Z_0/f\) 换回米，所以弱透视的立方体永远是同一个大小）。右图是最大偏差（占立方体边长的百分比）随 \(Z_0\) 的变化。

弱透视的另一个说法是投影雅可比的第三列趋于 0。完整的像素 \(\mathbf u = (u,v)\) 对相机系三维点 \(\mathbf X=(X,Y,Z)\) 的雅可比是 \(2\times3\) 矩阵（推导见第 5 节）：\(J = \tfrac1Z\begin{bmatrix}f_x&0&-f_xX/Z\\0&f_y&-f_yY/Z\end{bmatrix}\)。前两列是"横向移动 1 m，像素动 \(f/Z\) px"，第三列 \(-f X/Z^2\) 是"靠近或远离时像素沿径向外扩"，\(Z\) 大时可以忽略。它是 \(2\times3\)，必有一维零空间，这一维就是视线方向 \(\mathbf X\) 本身（读数里验证 \(J\mathbf X=\mathbf 0\)）：沿射线移动三维点，像素纹丝不动，与 FIG 3 完全吻合。这个雅可比在视觉伺服里叫"图像雅可比"（interaction matrix 的平移部分），在光束法平差（11.6 节）里则是每个重投影误差对点坐标的导数。

When the depth relief \(\Delta Z\) of an object is much smaller than its mean distance \(Z_0\), the divisor \(Z\) in the projection formula is nearly the constant \(Z_0\) and projection becomes approximately linear: \(u\approx f X/Z_0 + c_x\), the weak-perspective model (scaled orthography): drop \(Z\) and scale everything by \(f/Z_0\). In the figure a cube sits at depth \(Z_0\); amber is the true perspective projection and the purple dashed cube is weak perspective (both are converted back to metres with \(Z_0/f\), so the weak-perspective cube always has the same size). The right panel plots the maximum deviation, as a percentage of the cube's edge, against \(Z_0\).

Another way to say weak perspective: the third column of the projection Jacobian tends to 0. The full Jacobian of the pixel \(\mathbf u = (u,v)\) with respect to the camera-frame 3D point \(\mathbf X=(X,Y,Z)\) is a \(2\times3\) matrix (derived in Section 5): \(J = \tfrac1Z\begin{bmatrix}f_x&0&-f_xX/Z\\0&f_y&-f_yY/Z\end{bmatrix}\). The first two columns say "a sideways move of 1 m shifts the pixel by \(f/Z\) px", while the third, \(-f X/Z^2\), says "moving closer or farther pushes the pixel radially outward or inward" and is negligible for large \(Z\). Being \(2\times3\) it must have a one-dimensional null space, and that dimension is the viewing direction \(\mathbf X\) itself (the readout checks \(J\mathbf X=\mathbf 0\)): move the 3D point along the ray and the pixel does not budge, exactly as in FIG 3. In visual servoing this is the "image Jacobian" (the translational part of the interaction matrix), and in bundle adjustment (Lesson 11.6) it is the derivative of every reprojection error with respect to the point.

机器人常常贴一个尺寸已知的方形标记（AprilTag、ArUco），用它既能认出"这是哪个物体"，又能直接估计距离。下面的实验是这个思路的蒙特卡洛版：对 0.5 到 10 m 之间等间隔的 250 个距离，用 \(\tilde s = fS/Z + \) 噪声 测得像长，再用 \(\hat Z = fS/\tilde s\) 估计距离。测量噪声取 \(\sigma_s = \sqrt2\,\sigma\)（一条边的长度由两个角点各带噪声 \(\sigma\) 的位置之差得到）。绿色带是线性化预测的 \(\pm1\sigma\) 范围 \(Z^2\sigma_s/(fS)\)：近处几乎看不出误差，远处散点张成喇叭口。

这个模型是简化的：真实的标记检测器用角点的亚像素位置，并用四个角点通过 PnP（11.4 节）求完整的 6 自由度位姿，不要求标记正对相机；但误差随 \(Z^2\) 增长、与 \(fS\) 成反比的规律是一样的。所以要想看得远，办法只有三条：标记做大、焦距拉长（视场收窄）、噪声压低。

Robots often put a square marker of known size (AprilTag, ArUco) on things: it identifies "which object this is" and directly gives a distance estimate. The experiment below is a Monte Carlo version of that idea: for 250 evenly spaced distances from 0.5 to 10 m, measure the image length as \(\tilde s = fS/Z + \) noise and estimate the distance as \(\hat Z = fS/\tilde s\). The measurement noise is \(\sigma_s = \sqrt2\,\sigma\) (an edge length is the difference of two corner positions, each with noise \(\sigma\)). The green band is the linearized \(\pm1\sigma\) prediction \(Z^2\sigma_s/(fS)\): negligible error nearby, a trumpet of scatter far away.

The model is simplified: real marker detectors use sub-pixel corner positions and recover the full 6-DoF pose from the four corners by PnP (Lesson 11.4), without requiring the tag to face the camera; but the law is the same: error grows like \(Z^2\) and is inversely proportional to \(fS\). So to see farther there are only three levers: a bigger tag, a longer focal length (narrower view), or lower noise.