Part III · 第 11 章 多视图几何 · 11.2Part III · Ch. 11 Multi-View Geometry · 11.2

立体视觉与深度估计Stereo Vision & Depth

一台相机把三维世界压成二维图像，深度被丢掉了；两台相机并排放，同一个点在两幅图里的位置差（视差）就把深度还原出来，公式只有一行：\(Z=fB/d\)。真正难的是找到匹配，而物理上的代价是：深度误差随距离的平方增长。本节用随机点立体图（random-dot stereogram）亲手跑块匹配、左右一致性检验和半全局匹配（SGM），看它们在纹理缺失、重复纹理、遮挡处怎样失败；再看误差为什么是 \(Z^2\)，以及怎样用一个纯旋转把两台随便摆放的相机"矫正"成可以逐行搜索的标准形式。 One camera flattens the 3D world and throws depth away; put two side by side and the offset of the same point between the two images, the disparity, gives depth back in a single line: \(Z=fB/d\). The hard part is finding the match, and the price set by physics is that depth error grows with the square of distance. In this lesson you run block matching, the left-right consistency check and semi-global matching (SGM) live on a random-dot stereogram, watch where they fail (no texture, repeated texture, occlusion), see why the error is \(Z^2\), and rectify two arbitrarily placed cameras with a pure rotation so that matching becomes a row-by-row search.

约 35 分钟~35 min 前置：4.5 不确定性传播 · 10.1 针孔相机 · 10.3 内参 \(K\) · 11.1 对极几何Prereqs: 4.5 Propagating Uncertainty · 10.1 Pinhole · 10.3 Intrinsics \(K\) · 11.1 Epipolar Geometry 5 个交互5 interactives

    1为什么机器人需要它Why robots need this

    机器人要抓东西、躲障碍、落脚，需要的不是"图像里有什么"，而是"它在哪、有多远"。单目图像做不到这一点：10.1 节的针孔投影 \(u=fX/Z+c_x\) 把 \((X,Z)\) 揉成了一个数 \(u\)，沿着同一条视线的所有点成像在同一个像素。人有两只眼睛，机器人也一样：ZED、Bumblebee 这样的双目相机，RealSense D4xx 这样带红外投射器的主动双目，人形机器人头部的一对镜头，无人机前向的双目避障，火星车的导航相机，都是在用两个视角换回丢失的一维。

    • 几何是一行公式：并排放的两台相机（"矫正"之后），同一点在两幅图里水平错开 \(d\) 个像素，深度 \(Z=fB/d\)（\(B\) 是基线，\(f\) 是以像素计的焦距）。
    • 难点是匹配：在右图里找到左图那个点的"同一点"。11.1 节告诉我们，矫正之后对极线就是水平扫描线，匹配退化成一维搜索；但纹理缺失、重复纹理、遮挡和噪声会让这个一维搜索出错。
    • 物理给出的代价：深度误差 \(\sigma_Z\approx Z^2\sigma_d/(fB)\)，越远越糟，而且糟得很快（4.5 节的误差传播）。基线、焦距、视差精度共同决定一台双目相机能"看多远"。

    本节把这三件事各做成一个实验：FIG 1 是几何，FIG 2 是匹配，FIG 3 是误差；FIG 4 说明怎样把两台任意摆放的相机变成 FIG 1 的标准形式；FIG 5 把视差图变回三维点云。

    To grasp, to avoid, to step, a robot needs not "what is in the image" but "where is it and how far". One camera cannot say: the pinhole projection \(u=fX/Z+c_x\) of Lesson 10.1 squeezes \((X,Z)\) into one number \(u\), and every point on a viewing ray lands on the same pixel. Humans have two eyes and so do robots: ZED and Bumblebee stereo cameras, active-stereo RealSense D4xx units with an infrared projector, the pair of lenses in a humanoid head, the forward stereo pair of a drone's obstacle avoidance, the navigation cameras of Mars rovers all trade two viewpoints for the lost dimension.

    • The geometry is one line: for two cameras side by side (after "rectification"), the same point is shifted horizontally by \(d\) pixels between the images, and its depth is \(Z=fB/d\) (\(B\) the baseline, \(f\) the focal length in pixels).
    • The hard part is matching: finding in the right image the "same point" as in the left. Lesson 11.1 says that after rectification epipolar lines are horizontal scanlines, so matching becomes a 1-D search; but missing texture, repeated texture, occlusion and noise make that 1-D search go wrong.
    • The physics sets the price: depth error \(\sigma_Z\approx Z^2\sigma_d/(fB)\), worse with distance and worse fast (error propagation, Lesson 4.5). Baseline, focal length and disparity precision decide how far a stereo camera "can see".

    Each gets its own experiment: FIG 1 is the geometry, FIG 2 the matching, FIG 3 the error; FIG 4 shows how two arbitrarily placed cameras are turned into FIG 1's standard form; FIG 5 turns a disparity map back into a 3D point cloud.

    2直觉：视差 = 基线被"缩短"了多少Intuition: disparity is how much the baseline shrinks

    伸出一根手指，交替闭上左右眼：近处的手指"跳"得很厉害，远处的墙几乎不动。这个"跳动"就是视差，而且越近越大，无穷远为 0。把它画成俯视图：两台相机相距 \(B\)，一个点在深度 \(Z\) 处；两条视线在像平面上各留下一个标记，左右标记的水平位置之差就是 \(d\)。下图用相似三角形说明 \(d/f=B/Z\)：以两个相机中心和点 \(P\) 为顶点的大三角形，与以两个标记和 \(P\) 为顶点的小三角形相似，所以两个标记之间的距离是 \(B\) 缩小到 \((Z-f')/Z\) 倍（\(f'\) 是画出来的像平面距离），恰好是 \(B\) 少了 \(d\)（同样按绘图比例）。

    拖动琥珀色的点，或调基线 \(B\)、焦距 \(f\)、搜索范围 \(D\)（匹配算法最多愿意搜多少像素）。注意三件事：视差与 \(Z\) 成反比（右下曲线）；整数视差对应一层一层的深度，越远层越疏；点太近，视差超过 \(D\)，算法根本找不到它。（画图为了看清相似三角形，像平面放大绘制：1 像素画成 0.4 mm。）

    Hold up a finger and close your left and right eyes alternately: the near finger "jumps" a lot, the far wall hardly moves. That jump is disparity, and it is large when near and zero at infinity. In the top view below two cameras sit \(B\) apart and a point is at depth \(Z\); each viewing ray leaves a mark on its image plane and the horizontal difference of the two marks is \(d\). The figure shows similar triangles giving \(d/f=B/Z\): the big triangle (the two camera centres and the point \(P\)) is similar to the small one (the two marks and \(P\)), so the marks are \(B\) shrunk by the factor \((Z-f')/Z\) apart (\(f'\) is the drawn image-plane distance), which is exactly \(B\) minus \(d\) (also in drawing scale).

    Drag the amber point, or change the baseline \(B\), focal length \(f\) and search range \(D\) (the most pixels the matcher is willing to search). Notice three things: disparity is inversely proportional to \(Z\) (bottom-right curve); integer disparities are layers of depth, sparser the farther; and a point too near has disparity beyond \(D\), so the matcher cannot find it at all. (To make the similar triangles visible the image planes are drawn magnified: 1 pixel is drawn as 0.4 mm.)

    FIG 1视差 ⇔ 深度Disparity ⇔ depth
    俯视图：拖动 PTOP VIEW · drag P
    两幅图的同一行（像素）ONE ROW OF EACH IMAGE (px)
    深度 Z 随视差 d 变化DEPTH Z vs DISPARITY d
    左上：俯视图，相机朝上（\(+Z\)），蓝色是左相机、紫色是右相机，两条细线是视线，短横线是（放大绘制的）像平面，上面的小点是点 \(P\) 的像 \(u_L,u_R\)。琥珀色虚线连接两个标记，它与基线平行，长度是 \(B-d\)（按绘图比例）。右上：左右图的同一行，\(u_L\) 与 \(u_R\) 之差即视差 \(d\)。右下：\(Z=fB/d\) 的曲线，圆点是整数视差对应的深度层，红色区域是 \(d>D\)（搜索不到）。若点落在任一图像之外，立体就失效：只有两台相机视野的重叠区才能测深度。 Top left: top view, cameras looking up (\(+Z\)); blue is the left camera and purple the right one, thin lines are the viewing rays, short horizontal segments are the (magnified) image planes and the dots on them are the images \(u_L,u_R\) of the point \(P\). The dashed amber segment joins the two marks; it is parallel to the baseline and has length \(B-d\) (in drawing scale). Top right: one row of each image; \(u_L-u_R\) is the disparity \(d\). Bottom right: the curve \(Z=fB/d\); dots are the depth layers of integer disparities and the red zone is \(d>D\) (not found by the search). If the point leaves either image, stereo fails: depth is only measurable in the overlap of the two fields of view.
    试一试Try this
    • 把点从近处拖到离相机至少两倍远的地方：视差变成原来的一半。右下曲线上的点沿着哪条曲线移动？Drag the point from near to at least twice as far: the disparity halves. Which curve does the dot slide along in the bottom-right panel?
    • 保持点不动，把基线 \(B\) 调到 0.60 m：视差随之变成两倍。双目相机的"眼距"越大，同一深度的视差越大，深度分辨率越高，但近处更容易看不见（想想为什么）。Leave the point alone and raise the baseline \(B\) to 0.60 m: disparity doubles. A wider "eye distance" gives more disparity at the same depth and hence finer depth resolution, but near points get harder to see (why?).
    • 把点拖到足够近，使读数出现"太近：d > D"。最小可测深度 \(Z_{\min}=fB/D\)；想看得更近，要减小 \(B\)、减小 \(f\) 或增大搜索范围 \(D\)。Drag the point close enough that the readout says "too close: d > D". The minimum measurable depth is \(Z_{\min}=fB/D\); to see nearer, shrink \(B\), shrink \(f\), or enlarge the search range \(D\).
    核心洞见Key insight

    矫正之后，立体视觉有两个量互为倒数：视差 \(d\) 与深度 \(Z\) 成反比。视差在像素上是均匀的（匹配算法一次走一个像素），但深度不均匀：近处一个像素对应几毫米，远处一个像素对应几米。后面所有现象，深度误差随 \(Z^2\) 增长、远处分层、近处丢失，都是这一个反比关系的推论。

    After rectification stereo has two quantities that are reciprocals: disparity \(d\) is inversely proportional to depth \(Z\). Disparity is uniform in pixels (a matcher steps one pixel at a time) but depth is not: one pixel is a few millimetres near and several metres far. Everything that follows, depth error growing as \(Z^2\), far depths falling into layers, near points getting lost, is a consequence of this one reciprocal.

    3动手玩：随机点立体图与块匹配实验室Play: random-dot stereograms and the block-matching lab

    怎样"找同一点"？最朴素的做法是块匹配：取左图以 \((x,y)\) 为中心的一小块，在右图的同一行上从 \(x\) 往左一个像素一个像素地滑动，计算每个位置 \(d\) 的差异 \(C(x,y,d)\)（比如绝对差之和 SAD），选差异最小的 \(d\)（赢者通吃，WTA）。所有 \(C(x,y,d)\) 排成的三维数组叫代价体（cost volume）。

    为了知道算法对不对，我们需要真值。下图的两幅图是程序合成的随机点立体图（Julesz 1960 年代用它证明了"视差本身就足以产生深度感"）：一个倾斜的地面、一个悬浮的方块、一个圆顶，还有两块"困难区"——一面没有纹理的墙和一片周期条纹。右图由左图按真值视差前向变换得到（近处遮住远处），所以每个像素的真视差是已知的，右图里被遮住的左图像素无处可配（半遮挡）。下面跑的是真正的算法，不是预先画好的图：点击左图任一像素，看它的代价曲线 \(C(d)\)，看最小值落在哪里。

    How do we "find the same point"? The plainest way is block matching: take a small patch of the left image around \((x,y)\), slide it along the same row of the right image from \(x\) leftwards one pixel at a time, compute a dissimilarity \(C(x,y,d)\) at each candidate \(d\) (say the sum of absolute differences, SAD), and pick the \(d\) with the smallest cost (winner-take-all, WTA). All the \(C(x,y,d)\) together form a 3-D array called the cost volume.

    To know whether the algorithm is right we need ground truth. The two images below are a synthetic random-dot stereogram (Julesz used these in the 1960s to show that disparity alone is enough for depth perception): a tilted ground plane, a floating square, a dome, and two "hard zones", a textureless wall and a patch of periodic stripes. The right image is the left one forward-warped by the true disparity (near surfaces hide far ones), so every pixel's true disparity is known, and left pixels hidden in the right image have nothing to match (half-occlusion). What runs below is the real algorithm, not a pre-drawn picture: click any pixel of the left image to see its cost curve \(C(d)\) and where the minimum lands.

    FIG 2块匹配 / SGM 实验室Block-matching / SGM lab
    左图（点击选像素）LEFT · click a pixel
    右图（同一行）RIGHT · same row
    估计视差ESTIMATED DISPARITY
    左右图是同一个随机点图案的两个视角；估计视差与真值用同一色标（蓝 = 远 / 视差小，红 = 近 / 视差大）；第四幅可切换"误差"（绿：误差 ≤ 1 像素，红：更大，紫：被左右检验拒绝，暗灰：真值遮挡或超出右图，不参与统计）。底部图：代价曲线显示选中像素的 \(C(d)\)（绿色虚线是真值，琥珀色圆圈是赢者）；SGM 打开时紫线是聚合后的代价 \(S/4\)；"代价切片"显示该行全部 \(C(x,d)\)，青色谷底就是匹配，绿点是真值路径，琥珀点是估计。统计只算两个视角都看得见的像素；"精细误差"是误差 ≤ 1 像素的那些像素的平均绝对误差，亚像素拟合改善的正是它。窗口为 \((2r+1)\times(2r+1)\)；\(\lambda\) 即 SGM 的惩罚：\(P_2=\lambda,\ P_1=\lambda/4\)。 Left and right are two views of one random-dot pattern; the estimated disparity and the truth share one colour scale (blue = far / small disparity, red = near / large). The fourth panel switches to "error" (green: error ≤ 1 px, red: larger, purple: rejected by the left-right check, dim grey: occluded in the truth or outside the right image, not scored). Bottom plot: the cost curve shows \(C(d)\) of the selected pixel (dashed green is the truth, the amber circle the winner); with SGM on, the purple line is the aggregated cost \(S/4\); "cost slice" shows \(C(x,d)\) of the whole row, where the cyan valley is the match, green dots the true path and amber dots the estimate. Statistics only count pixels visible in both views; "fine error" is the mean absolute error of pixels within 1 px, the part sub-pixel fitting improves. The window is \((2r+1)\times(2r+1)\); \(\lambda\) is SGM's penalty: \(P_2=\lambda,\ P_1=\lambda/4\).
    试一试Try this
    • 把窗口半径 \(r\) 调到 0（1×1 窗口）：视差图满是噪点；再调到 6 以上（13×13）：平滑了，但物体边缘向外"胖"了一圈（前景膨胀）。找一个折中。Set the window radius \(r\) to 0 (1×1): the disparity map is pure speckle; then go beyond 6 (13×13): smooth, but object edges grow outwards (foreground fattening). Find a compromise.
    • 在左图里分别点击纹理缺失的墙（右上）和周期条纹（右下）：墙的代价曲线是一条平线（没有信息），条纹的曲线有周期性的多个同样深的谷（歧义）。Click the textureless wall (top right) and the periodic stripes (bottom right) in the left image: the wall's cost curve is flat (no information), the stripes' curve has several equally deep, evenly spaced valleys (ambiguity).
    • 把优化器切到 SGM：周期条纹区的错误大幅下降，因为平滑项把歧义的谷"拉"向邻居的答案；墙仍然是错的——平滑项不能凭空创造信息。Switch the optimizer to SGM: errors in the periodic stripes drop sharply, because the smoothness term pulls an ambiguous valley towards its neighbours' answer; the wall stays wrong, since smoothness cannot invent information.
    • 打开左右一致性检验：深度突变处左侧的半遮挡像素（紫色）被拒绝，统计中的错误率下降，代价是密度下降。Turn on the left-right consistency check: half-occluded pixels on the left side of depth jumps (purple) are rejected, the error rate falls, and density falls with it.
    • 把"右图垂直错位"调到 1 像素：匹配几乎全军覆没。矫正不准、标定偏一个像素，细纹理就再也配不上（11.1 节的对极线不再是同一行）。Raise the "right image vertical offset" to 1 px: matching all but collapses. With a rectification or calibration off by one pixel, fine texture no longer lines up (the epipolar line of Lesson 11.1 is no longer the same row).
    三种看法：窗口、代价体、平面扫描Three views: window, cost volume, plane sweep

    窗口：对每个像素独立地比较两小块。代价体：先算出所有 \(C(x,y,d)\)，再在体上做优化（WTA、SGM 乃至 3D 卷积网络都只是"在代价体上选一条路径"）。平面扫描：第 \(d\) 层代价 \(C(\cdot,\cdot,d)\) 就是把右图平移 \(d\) 个像素再和左图比，相当于假设整个场景是深度 \(Z=fB/d\) 的一张正对相机的平面；扫过所有 \(d\)，每个像素就挑出让它最一致的那张平面。FIG 2 的"代价切片"把一行的代价体直接画了出来。

    Window: compare two small patches independently per pixel. Cost volume: first compute every \(C(x,y,d)\), then optimise on the volume (WTA, SGM, even 3D-convolutional networks are all "choose a path through the cost volume"). Plane sweep: layer \(d\) of the volume, \(C(\cdot,\cdot,d)\), is the right image shifted by \(d\) pixels and compared with the left, i.e. it assumes the whole scene is a fronto-parallel plane at depth \(Z=fB/d\); sweep over all \(d\) and each pixel picks the plane that agrees with it best. FIG 2's "cost slice" draws one row of the cost volume directly.

    4另外两个角度：误差为什么是 Z²，以及矫正Two more views: why the error is Z², and rectification

    深度误差随距离的平方增长Depth error grows with the square of distance

    假设匹配算法找到的视差有一个小误差 \(\sigma_d\)（FIG 2 里亚像素之内的抖动）。\(Z=fB/d\) 对 \(d\) 求导得 \(\mathrm dZ/\mathrm dd=-fB/d^2=-Z^2/(fB)\)，这就是 4.5 节里的 Jacobian，于是 \(\sigma_Z\approx\frac{Z^2}{fB}\sigma_d\)。同样的 0.25 像素误差，在 2 m 处不到 1.5 cm，在 20 m 处超过 1 m，因为远处一个像素代表的深度间隔大得多。左图画 \(\sigma_Z(Z)\)，右图是蒙特卡洛：在每个深度站放一排点，给视差加高斯噪声再反推深度，散开的程度就是误差。数字是示意性的，用来比较趋势，不是某款相机的规格。

    Suppose the matcher's disparity has a small error \(\sigma_d\) (the sub-pixel jitter in FIG 2). Differentiating \(Z=fB/d\) gives \(\mathrm dZ/\mathrm dd=-fB/d^2=-Z^2/(fB)\), the Jacobian of Lesson 4.5, so \(\sigma_Z\approx\frac{Z^2}{fB}\sigma_d\). The same 0.25 px error is under 1.5 cm at 2 m and over 1 m at 20 m, because one pixel far away spans a much larger depth interval. The left plot draws \(\sigma_Z(Z)\); the right one is Monte Carlo: a row of points at each depth station, Gaussian noise added to the disparity and depth recomputed from it, so the spread is the error. The numbers are illustrative, meant for comparing trends, not the spec of any particular camera.

    FIG 3深度误差 ∝ Z²Depth error ∝ Z²
    σ_Z (cm) 随 Z：拖动圆圈选深度σ_Z (cm) vs Z · drag the circle
    蒙特卡洛：估计深度 Z (m)MONTE CARLO · estimated Z (m)
    粗琥珀线是当前参数的 \(\sigma_Z(Z)\)，细线是三个预设；绿色虚线是 ±5 cm 的要求，它与曲线的交点就是"±5 cm 能用到多远"。右图：每个深度站（灰）与你选的深度（琥珀）上各有一排点，横向位置随意，纵向是 \(\hat Z=fB/(d+n)\)，\(n\sim\mathcal N(0,\sigma_d^2)\)，虚线是真值。注意散布不对称：近处的点聚拢，远处向"更远"拖出长尾，因为 \(1/d\) 是凸函数。读数中的"蒙特卡洛"是 2000 次采样的标准差，与一阶公式只在远处才略有差别。 The thick amber line is \(\sigma_Z(Z)\) for the current parameters and the thin lines are the three presets; the dashed green line is the ±5 cm requirement, and where it crosses the curve is "how far ±5 cm works". Right: a row of points at each depth station (grey) and at your chosen depth (amber), lateral position arbitrary, vertical position \(\hat Z=fB/(d+n)\) with \(n\sim\mathcal N(0,\sigma_d^2)\), dashed lines are the truth. Note the spread is lopsided: it is tight on the near side and has a long tail towards "farther", because \(1/d\) is convex. "Monte Carlo" in the readout is the standard deviation of 2000 samples; it differs from the first-order formula only far away.
    试一试Try this
    • 把深度圆圈（或"深度 Z"滑块）从起点 6 m 拖到两倍远（12 m 以上）或一半近（3 m 以下）：误差变成 4 倍（或 1/4）。读数里的相对误差验证了平方律。Drag the depth circle (or the "depth Z" slider) from its start at 6 m to twice the distance (12 m or more) or half (3 m or less): the error becomes 4× (or ¼×). The relative error in the readout confirms the square law.
    • 调基线、焦距和视差精度，让"±5 cm 可用距离"达到 10 m 以上。需要的 \(fB/\sigma_d\) 是多少？哪个旋钮最便宜？（提示：焦距 = 视场角，基线 = 体积和近距盲区，\(\sigma_d\) = 算法与标定。）Tune baseline, focal length and disparity precision until the "±5 cm range" reaches 10 m or more. What value of \(fB/\sigma_d\) does that need, and which knob is cheapest? (Hint: focal length = field of view, baseline = size and near-range blind zone, \(\sigma_d\) = algorithm and calibration.)

    矫正：把任意摆放的两台相机变成标准形式Rectification: turning two arbitrary cameras into the standard form

    FIG 1 和 FIG 2 假设两台相机光轴平行、共面、像平面在同一平面上。真实的两台相机永远不会这样：装配有微小的转角，镜头也不同。11.1 节说，对极线一般是斜的，每个点要沿一条不同的斜线搜索。矫正（rectification）是在不动相机中心的前提下，把两台相机各自转到同一个新朝向，使新的 \(x\) 轴就是基线方向：这时基线在新坐标系里是 \((B,0,0)\)，对极线自动变成同一行。因为只改变朝向，新图像和旧图像之间只差一个由旋转诱导的单应（10.5 节）\(H=K\,R_{\text{new}}^{\top}R_i\,K^{-1}\)。

    下图的右相机有可调的偏航、俯仰、滚转。拖动左图中的琥珀点选一个像素，右图里画出它的对极线（它在右图里所有可能的位置），再调"试探深度"，对应的点会在这条线上滑动。把"矫正程度" \(t\) 从 0 拉到 1，看两台相机怎样转向同一个朝向，斜线变成水平线，读数里同名点的行差缩小到数值误差 \(10^{-10}\) 像素以下。

    FIG 1 and FIG 2 assumed two cameras with parallel optical axes in a common image plane. Real pairs never are: assembly leaves small rotations. Lesson 11.1 says epipolar lines are generally slanted, so every point has to be searched along a different slanted line. Rectification keeps the camera centres fixed and rotates each camera to the same new orientation whose \(x\)-axis is the baseline direction: the baseline is then \((B,0,0)\) in the new frame and the epipolar lines are automatically the same row. Because only orientation changes, the new and old images differ by a rotation-induced homography (Lesson 10.5), \(H=K\,R_{\text{new}}^{\top}R_i\,K^{-1}\).

    The right camera below has adjustable yaw, pitch and roll. Drag the amber point in the left image to choose a pixel; the right image draws its epipolar line (every place it can be in the right image), and the "trial depth" slider moves the matching point along that line. Pull the "rectification" amount \(t\) from 0 to 1 and watch both cameras turn to a common orientation, the slanted lines become horizontal, and the row difference of matching points shrink to numerical round-off below \(10^{-10}\) px.

    FIG 4矫正Rectification
    左图：拖动琥珀点LEFT · drag the amber point
    右图：对极线RIGHT · epipolar line
    三维：相机、基线与新坐标轴（拖动旋转）3D · cameras, baseline and new axes (drag to orbit)
    场景是两个盒子和地面网格，相机内参相同（\(f=420\) px，640×480）。琥珀点是左图中选的像素，沿它的视线取试探深度，投影到右图得到空心圆环；蓝线是对极线，上面的小圆点是 1、2、…、8 m 深度的位置（越远越挤，对应 FIG 1 的反比）。三维图中，半透明三角形是对极平面 \(C_L,C_R,X\)；淡色的视锥是矫正前的相机，实线是 \(t\) 时刻的相机；红绿蓝箭头是新坐标轴 \(\mathbf r_1,\mathbf r_2,\mathbf r_3\)（\(\mathbf r_1\) = 基线方向）。"行差"是 16 个盒子角点的 \(|v_L-v_R|\) 的最大值；同时用单应 \(H_L,H_R\) 直接变换原始像素再核对一遍。 The scene is two boxes and a ground grid; both cameras have the same intrinsics (\(f=420\) px, 640×480). The amber point is the pixel chosen in the left image; a trial depth along its viewing ray, projected into the right image, gives the open ring; the blue line is the epipolar line and the small dots on it are the positions for depths 1, 2, …, 8 m (crowding together with distance, FIG 1's reciprocal). In the 3D view the translucent triangle is the epipolar plane \(C_L,C_R,X\); faint frustums are the original cameras and solid ones the cameras at amount \(t\); the red, green and blue arrows are the new axes \(\mathbf r_1,\mathbf r_2,\mathbf r_3\) (\(\mathbf r_1\) = baseline direction). "Row difference" is the largest \(|v_L-v_R|\) over the 16 box corners; the original pixels are also pushed through the homographies \(H_L,H_R\) directly as a cross-check.
    试一试Try this
    • 把"矫正程度"拉到 1（或点"矫正"按钮）：对极线变成水平线，行差降到 \(10^{-10}\) 像素以下。Pull the rectification amount to 1 (or press "Rectify"): the epipolar line turns horizontal and the row difference drops below \(10^{-10}\) px.
    • 矫正后拖动"试探深度"：圆环沿水平线移动，视差 \(d=u_L-u_R\) 与 \(fB/Z\) 完全一致。现在匹配只需要在这一行上搜索。After rectifying, drag the "trial depth": the ring slides along a horizontal line and the disparity \(d=u_L-u_R\) equals \(fB/Z\) exactly. Matching now only has to search this one row.

    5再看数学The math

    提示：把鼠标悬停在公式中带底色的符号上，FIG 1、FIG 2 或 FIG 3 中对应的元素会被高亮。Tip: hover the highlighted symbols in the formulas; the matching element in FIG 1, FIG 2 or FIG 3 lights up.

    矫正后的几何：Z = fB/d，以及由视差图恢复三维点Rectified geometry: Z = fB/d, and 3D points from a disparity map

    取左相机坐标系为参考（OpenCV 光学坐标系：\(x\) 右，\(y\) 下，\(z\) 前）。矫正后右相机相对左相机只沿 \(x\) 平移 \(B\)，内参相同。点 \(P=(X,Y,Z)\) 在右相机坐标系里是 \((X-B,Y,Z)\)，于是

    Take the left camera frame as reference (OpenCV optical frame: \(x\) right, \(y\) down, \(z\) forward). After rectification the right camera is the left one translated by \(B\) along \(x\) only, with the same intrinsics. The point \(P=(X,Y,Z)\) is \((X-B,Y,Z)\) in the right camera frame, so

    $$ u_L=c_x+f\frac{X}{Z},\quad u_R=c_x+f\frac{X-B}{Z},\quad v_L=v_R=c_y+f\frac{Y}{Z}\ \Longrightarrow\ \hl{d}{d}=u_L-u_R=\frac{\hl{f}{f}\,\hl{B}{B}}{\hl{Z}{Z}},\qquad Z=\frac{fB}{d}. $$

    视差永远 \(\ge0\)（左图为参考，右图里的点向左移）。反过来，视差图上每个像素 \((u,v,d)\) 都是一个三维点：

    Disparity is always \(\ge0\) (left image as reference; points move left in the right image). Conversely every pixel \((u,v,d)\) of a disparity map is a 3D point:

    $$ \begin{bmatrix}X\\Y\\Z\end{bmatrix}=\frac{B}{d}\begin{bmatrix}u-c_x\\ v-c_y\\ f\end{bmatrix},\qquad \begin{bmatrix}X\\Y\\Z\\W\end{bmatrix}=\underbrace{\begin{bmatrix}1&0&0&-c_x\\0&1&0&-c_y\\0&0&0&f\\0&0&1/B&0\end{bmatrix}}_{Q}\begin{bmatrix}u\\v\\d\\1\end{bmatrix}\ \text{(then divide by } W=d/B\text{)}. $$

    右边的 \(4\times4\) 矩阵 \(Q\) 就是 OpenCV 的 stereoRectify 返回、reprojectImageTo3D 使用的那个重投影矩阵（符号约定略有不同）。注意 \(d\) 的单位是像素，\(f\) 也是像素，\(B\) 才是米，所以 \(Z\) 的单位是米。

    The \(4\times4\) matrix \(Q\) is the reprojection matrix returned by OpenCV's stereoRectify and used by reprojectImageTo3D (up to sign conventions). Both \(d\) and \(f\) are in pixels and only \(B\) is in metres, so \(Z\) comes out in metres.

    误差模型The error model

    $$ \frac{\partial Z}{\partial d}=-\frac{fB}{d^{2}}=-\frac{Z^{2}}{fB}\ \Longrightarrow\ \hl{sz}{\sigma_Z}\approx\frac{Z^{2}}{fB}\,\hl{sd}{\sigma_d},\qquad \frac{\sigma_Z}{Z}=\frac{\sigma_d}{d},\qquad \sigma_X\approx\frac{Z}{f}\sigma_u\ (\text{lateral, only linear in } Z). $$

    第一个式子是 4.5 节的 \(\sigma_Z^2=J\sigma_d^2J\) 在标量情形的样子。三个推论：①误差 \(\propto Z^2\)；②相对误差等于视差的相对误差，视差只有几个像素时一个像素就是几十个百分点；③横向误差只随 \(Z\) 线性增长，所以双目是"横向准、深度粗"。一阶近似在 \(\sigma_d\ll d\) 时有效；\(d\) 接近 \(\sigma_d\) 时分布严重偏斜（\(\mathbb E[1/d]>1/\mathbb E[d]\)，FIG 3 右图的长尾），甚至 \(d+n\le0\) 给出"负深度"。整数视差给出的深度层间隔是 \(\Delta Z\approx Z^2/(fB)\)，很粗；OpenCV 的 StereoBM / StereoSGBM 因此输出带 4 位小数的定点视差（要除以 16）。另外 \(Z\propto fB\)：标定里 \(f\) 或 \(B\) 的百分之一的误差，就是整个深度图百分之一的系统尺度误差。

    The first line is Lesson 4.5's \(\sigma_Z^2=J\sigma_d^2J\) in the scalar case. Three consequences: ① error \(\propto Z^2\); ② the relative error equals the relative disparity error, so with a disparity of a few pixels one pixel of error is tens of percent; ③ lateral error grows only linearly with \(Z\), so stereo is "accurate sideways, coarse in depth". The first-order result holds for \(\sigma_d\ll d\); when \(d\) approaches \(\sigma_d\) the distribution is badly skewed (\(\mathbb E[1/d]>1/\mathbb E[d]\), the long tail in FIG 3) and \(d+n\le0\) even yields "negative depth". Integer disparities give depth layers spaced \(\Delta Z\approx Z^2/(fB)\) apart, which is coarse; OpenCV's StereoBM / StereoSGBM therefore output fixed-point disparity with four fractional bits (divide by 16). Also \(Z\propto fB\): a 1 % error in \(f\) or \(B\) from calibration is a 1 % systematic scale error of the whole depth map.

    匹配代价、聚合与选择Matching cost, aggregation and selection

    $$ \hl{cost}{C(x,y,d)}=\frac{1}{(2r+1)^2}\sum_{|i|,|j|\le r}\big|I_L(x{+}i,y{+}j)-I_R(x{-}d{+}i,y{+}j)\big|\ \ (\text{SAD}),\qquad \hat d(x,y)=\arg\min_{d\in[0,D]}C(x,y,d)\ \ (\text{WTA}). $$

    把绝对值换成平方得 SSD；换成 \(1-\mathrm{NCC}\)（减去窗口均值、除以窗口标准差，FIG 2 中的 NCC 就是 \((1-\mathrm{NCC})/2\)）就对左右相机的亮度偏置、增益（"曝光不一致"）不敏感；census 变换把窗口里每个像素与中心比大小，得到一个比特串，用汉明距离作代价，对任何单调的亮度变化都不变，硬件流水线里很常见。窗口大小是偏差—方差权衡：小窗口定位准但噪声大、纹理不足时无信息；大窗口平滑，但窗口跨过深度边缘时两侧的深度互相污染（前景膨胀）。

    Replacing the absolute value by a square gives SSD; replacing it by \(1-\mathrm{NCC}\) (subtract the window mean, divide by the window standard deviation; FIG 2's NCC is \((1-\mathrm{NCC})/2\)) makes the cost insensitive to brightness offset and gain between the two cameras ("exposure mismatch"); the census transform compares each window pixel with the centre to make a bit string and uses the Hamming distance, which is invariant to any monotone brightness change and common in hardware pipelines. Window size is a bias-variance trade-off: a small window localises well but is noisy and uninformative without texture; a large one smooths, but a window straddling a depth edge mixes the depths of both sides (foreground fattening).

    $$ \delta=\frac{C(\hat d-1)-C(\hat d+1)}{2\,\big[C(\hat d-1)-2C(\hat d)+C(\hat d+1)\big]}\ \ (\text{sub-pixel: } d^\star=\hat d+\delta),\qquad \text{accept } p \iff \big|d_L(p)-d_R\big(p-d_L(p)\big)\big|\le 1 . $$

    第一式是过最小值及两个邻居拟合抛物线的顶点（\(|\delta|\le\tfrac12\)）；它在 SAD 下有"像素锁定"偏差，所以只能当作改进，不是魔法。第二式是左右一致性检验：用同一个代价体，也对右图的像素 \(x'\) 选 \(d_R(x')=\arg\min_dC(x'+d,y,d)\)；如果左像素 \(p\) 的匹配像素回指的不是 \(p\)，说明这个匹配不可信，半遮挡像素几乎都过不了这一关。

    The first is the vertex of the parabola through the minimum and its two neighbours (\(|\delta|\le\tfrac12\)); with SAD it has "pixel-locking" bias, so it is an improvement, not magic. The second is the left-right consistency check: using the same cost volume, also pick \(d_R(x')=\arg\min_dC(x'+d,y,d)\) for each right pixel \(x'\); if the pixel a left pixel \(p\) matches does not point back to \(p\), the match cannot be trusted, and half-occluded pixels almost never pass.

    半全局匹配（SGM）Semi-global matching (SGM)

    逐像素 WTA 没有"相邻像素视差应当相近"的先验。理想的全局能量是 \(E(D)=\sum_p\big[C(p,D_p)+\sum_{q\in\mathcal N_p}P_1[|D_p-D_q|=1]+P_2[|D_p-D_q|>1]\big]\)：小跳变罚 \(P_1\)（倾斜面），大跳变罚 \(P_2\)（深度边缘）。二维网格上它是 NP 难的；Hirschmüller 的办法是沿 \(r\) 个一维方向做精确的动态规划，再把各方向的结果相加：

    Per-pixel WTA has no prior that neighbouring pixels have similar disparity. The ideal global energy is \(E(D)=\sum_p\big[C(p,D_p)+\sum_{q\in\mathcal N_p}P_1[|D_p-D_q|=1]+P_2[|D_p-D_q|>1]\big]\): small jumps cost \(P_1\) (slanted surfaces), big jumps \(P_2\) (depth edges). On a 2-D grid it is NP-hard; Hirschmüller's trick is exact dynamic programming along \(r\) one-dimensional paths and summing the results:

    $$ L_{r}(p,d)=C(p,d)+\min\!\Big(L_r(p{-}r,d),\ L_r(p{-}r,d{\pm}1)+P_1,\ \min_iL_r(p{-}r,i)+P_2\Big)-\min_iL_r(p{-}r,i),\qquad S(p,d)=\sum_rL_r(p,d),\ \ \hat d=\arg\min_dS. $$

    最后减去 \(\min_iL_r(p-r,i)\) 只是防止数值无限增长。每条路径的代价是 \(O(WHD)\)，FIG 2 用上下左右四条路径，原论文用 8 或 16 条，并用互信息代价而不是 SAD。OpenCV 的 StereoSGBM 是它的一个变体。SGM 之后，学习式立体（GC-Net、PSMNet 用 3D 卷积在特征代价体上做同样的事；RAFT-Stereo 用迭代更新；FoundationStereo 一类大模型追求零样本泛化）并没有改变框架：构造代价体，然后选一条路径，只是把代价和平滑先验换成了学出来的。

    Subtracting \(\min_iL_r(p-r,i)\) merely keeps numbers from growing without bound. Each path costs \(O(WHD)\); FIG 2 uses four paths (left, right, up, down), the paper uses 8 or 16 and a mutual-information cost instead of SAD, and OpenCV's StereoSGBM is a variant. Learned stereo (GC-Net and PSMNet run 3D convolutions on a feature cost volume; RAFT-Stereo iterates updates; large models in the FoundationStereo line aim at zero-shot generalisation) does not change the frame: build a cost volume, then choose a path through it, with the cost and the smoothness prior learned instead of hand-made.

    矫正的构造（纯旋转版）Constructing the rectification (rotation-only version)

    $$ \mathbf r_1=\frac{C_R-C_L}{\|C_R-C_L\|},\quad \mathbf r_2=\frac{\mathbf k_L\times\mathbf r_1}{\|\mathbf k_L\times\mathbf r_1\|},\quad \mathbf r_3=\mathbf r_1\times\mathbf r_2,\quad R_{\text{new}}=\begin{bmatrix}\mathbf r_1&\mathbf r_2&\mathbf r_3\end{bmatrix},\qquad H_i=K\,R_{\text{new}}^{\top}R_i\,K^{-1}. $$

    \(C_L,C_R\) 是相机中心（世界坐标），\(R_i\)（\({}^{w}R_{c}\)，相机到世界）是各自的朝向，\(\mathbf k_L\) 是左相机的光轴。新 \(x\) 轴取基线，新 \(y\) 轴取尽量接近原图"向下"的方向，新 \(z\) 轴补成右手系。验证：\(R_{\text{new}}^{\top}(C_R-C_L)=(\|C_R-C_L\|,0,0)^{\top}\)，所以两个相机在新坐标系里只差一个沿 \(x\) 的平移 \(B=\|C_R-C_L\|\)，于是 \(v_L=v_R\)。这就是 Fusiello–Trucco–Verri 的"紧凑算法"的核心；Bouguet 的做法（OpenCV stereoRectify 采用）把两台相机之间的相对旋转各转一半以减小畸变。FIG 4 的"矫正程度" \(t\) 用 \(R_i\exp(t\log(R_i^{\top}R_{\text{new}}))\) 插值（5.6 节的指数/对数映射）。真实系统中两台相机的 \(K\) 和畸变也不同：先去畸变，再用同一个新的 \(K\) 重采样。

    \(C_L,C_R\) are the camera centres (world frame), \(R_i\) (\({}^{w}R_{c}\), camera to world) the orientations and \(\mathbf k_L\) the left optical axis. The new \(x\)-axis is the baseline, the new \(y\)-axis is as close as possible to the old image "down", and the new \(z\)-axis completes a right-handed frame. Check: \(R_{\text{new}}^{\top}(C_R-C_L)=(\|C_R-C_L\|,0,0)^{\top}\), so in the new frame the two cameras differ by a translation along \(x\) alone, \(B=\|C_R-C_L\|\), hence \(v_L=v_R\). This is the core of the Fusiello-Trucco-Verri "compact algorithm"; Bouguet's method (used by OpenCV stereoRectify) splits the relative rotation between the cameras in half to reduce distortion. FIG 4's amount \(t\) interpolates with \(R_i\exp(t\log(R_i^{\top}R_{\text{new}}))\) (the exp/log maps of Lesson 5.6). In real systems the two cameras also have different \(K\) and distortion: undistort first, then resample with one common new \(K\).

    6机器人中的它：从视差图到点云，以及各种深度传感器In robotics: from disparity map to point cloud, and depth sensors compared

    视差图本身对机器人没用，要变成三维点才能避障、建图、规划抓取。下图用 FIG 2 当前的视差图，按 5 节的公式 \((X,Y,Z)=\frac{B}{d}(u-c_x,\,v-c_y,\,f)\) 逐像素重投影（假设 \(f=150\) px、\(B=0.12\) m，一个头部尺度的双目）。拖动旋转：地面是一张斜面，方块和圆顶悬在它前面。它直接显示出两件事：整数视差把点云切成一层一层的"纸板"（回到 FIG 1 的深度层），打开 FIG 2 的亚像素拟合并把真值设为亚像素就会变平滑；错误匹配（墙、遮挡处）变成飘在空中的"飞点"，下游算法必须过滤它们（左右检验、置信度、离群点滤波）。

    A disparity map is of no use to a robot until it becomes 3D points for obstacle avoidance, mapping and grasp planning. The figure takes FIG 2's current disparity map and reprojects every pixel with the formula of Section 5, \((X,Y,Z)=\frac{B}{d}(u-c_x,\,v-c_y,\,f)\) (assuming \(f=150\) px and \(B=0.12\) m, a head-sized stereo rig). Drag to orbit: the ground is a slanted plane, with the square and the dome floating in front of it. It shows two things directly: integer disparities slice the cloud into "cardboard" layers (FIG 1's depth layers), which smooth out when you turn on sub-pixel fitting in FIG 2 and set the truth to sub-pixel; and wrong matches (the wall, occlusions) become "flying pixels" in mid-air that downstream code must filter out (left-right check, confidence, outlier removal).

    FIG 5视差图 → 点云Disparity map → point cloud
    相机坐标系：x 右，z 前，y 向下（此处画成 z 向上）；颜色 = 视差camera frame: x right, z forward, y down (drawn z-up); colour = disparity
    每两个像素取一个点（4800 个点），被左右检验拒绝或视差为 0 的像素不画。回到 FIG 2 改变窗口、噪声、优化器、亚像素和遮挡检验，点云会同步更新；可以试试在"困难区"场景下关掉左右检验：墙和条纹的错点会飘到相机前后。 One point per two pixels (4800 points); pixels rejected by the left-right check or with zero disparity are not drawn. Go back to FIG 2 and change the window, noise, optimizer, sub-pixel and consistency check; the cloud follows. Try the "hard zones" scene with the left-right check off: wrong points from the wall and the stripes float in front of and behind the camera.

    深度传感器对比Depth sensors compared

    "主动"的意思是传感器自己发光。FIG 2 的纹理缺失的墙最能说明为什么要有主动双目：RealSense D4xx 的红外投射器向场景打一片伪随机点，它本质上就是 FIG 2 里那种随机点图案，墙上有了纹理，块匹配又能工作了（在 FIG 2 的场景选项里选"+ 红外点投射"，墙和条纹区的错误率会立刻降下来）。下表只给通用结论，具体量程与精度因型号而异。

    "Active" means the sensor emits light itself. FIG 2's textureless wall is the best argument for active stereo: the infrared projector of a RealSense D4xx casts a pseudo-random dot pattern on the scene, essentially the random-dot pattern of FIG 2, so the wall has texture and block matching works again (pick "+ IR dot projector" in FIG 2's scene options and the error rate of the wall and stripe zones drops at once). The table gives only general statements; actual range and accuracy vary by model.

    传感器Sensor 原理Principle 量程与误差随距离Range and error vs distance 强项 / 典型失败Strengths / typical failures
    被动双目（ZED、Bumblebee、机器人头部双目）Passive stereo (ZED, Bumblebee, robot-head pairs) 两台相机，匹配场景自身纹理Two cameras, match the scene's own texture 量程随基线增长，从几十厘米到几十米；误差 \(\propto Z^2\)Range grows with baseline, from tens of cm to tens of m; error \(\propto Z^2\) 室外阳光下可用、无需发光、分辨率高；纹理缺失的墙、反光/透明表面、重复纹理、夜间失败Works in sunlight, emits nothing, high resolution; fails on textureless walls, reflective or transparent surfaces, repeated texture and at night
    主动双目（RealSense D415/D435/D455）Active stereo (RealSense D415/D435/D455) 两台红外相机 + 红外点投射器，给场景加纹理Two IR cameras plus an IR dot projector that adds texture 室内常见量程约零点几米到数米；误差仍 \(\propto Z^2\)Typically some tenths of a metre to a few metres indoors; error still \(\propto Z^2\) 白墙也有深度；强日光会淹没投射图案（退回被动双目），黑色/镜面/玻璃仍然困难Plain walls get depth; strong sunlight washes out the pattern (falls back to passive stereo); black, mirror-like and glass surfaces remain hard
    结构光（Kinect v1、PrimeSense、手机人脸识别）Structured light (Kinect v1, PrimeSense, phone face unlock) 一台相机 + 已知图案的投射器：投射器相当于"虚拟第二台相机"One camera plus a projector of a known pattern: the projector acts as the "virtual second camera" 近距离（零点几米到数米）；误差随距离同样平方增长Short range (tenths of a metre to a few metres); error also grows quadratically 室内精度好；阳光、多台设备互相干扰、黑色表面Good indoors; sunlight, interference between several units, black surfaces
    ToF 相机（Kinect v2 / Azure Kinect 等）ToF camera (Kinect v2 / Azure Kinect and others) 调制红外光，测每个像素返回光的相位 / 飞行时间Modulated IR light; measures per-pixel phase / flight time of the return 室内约数米；误差增长比双目慢，不是 \(Z^2\)A few metres indoors; error grows more slowly than stereo, not as \(Z^2\) 无需匹配、白墙也行；多径干扰、物体边缘的"飞点"、强日光、深色吸光表面No matching needed, plain walls fine; multipath interference, "flying pixels" at object edges, strong sunlight, dark absorbing surfaces
    LiDAR（机械旋转、固态）LiDAR (spinning, solid-state) 发射激光脉冲，测往返时间Emits laser pulses, measures round-trip time 远（数十到数百米），厘米级误差基本不随距离变大Long (tens to hundreds of metres), centimetre-level error nearly independent of range 室外远距离、测距准；点稀疏、昂贵，玻璃/镜面/强吸光表面会漏掉或给出假点，没有颜色纹理Long outdoor range and accurate; sparse and costly, glass, mirrors and strongly absorbing surfaces drop out or give false points, no colour or texture
    真实系统里In real systems

    硬件：Stereolabs ZED 系列和 FLIR/Point Grey Bumblebee 是被动双目；Intel RealSense D415/D435/D455 是主动红外双目，深度由芯片内的专用处理器计算（Keselman 等，2017）；Luxonis OAK-D 在自带的视觉处理器上跑双目匹配；火星车的导航相机与避险相机，以及一些汽车的前视系统（如 Subaru EyeSight）都是双目对。软件：OpenCV 的 stereoRectify、StereoBM、StereoSGBM、reprojectImageTo3D 是本节全部流程的现成版本；ROS 的 stereo_image_proc 把它们接进机器人系统；GPU 上有 libSGM、NVIDIA VPI，以及 Isaac ROS 里的学习式双目视差网络；得到的点云再交给 Open3D / PCL（12.1 节）。机器人用法：无人机前向双目避障、人形机器人头部相机取抓取点云（9.3 节的抓取规划）、移动机器人的局部代价地图；视觉里程计与 SLAM 用双目可以直接得到尺度（Part IV 回来讲）。深度补全（depth completion）把稀疏但可靠的测距（LiDAR、ToF）和稠密的图像融合，补上主动传感器丢失的区域。

    Hardware: the Stereolabs ZED family and FLIR/Point Grey Bumblebee are passive stereo; Intel RealSense D415/D435/D455 are active IR stereo whose depth is computed by a dedicated on-chip processor (Keselman et al., 2017); the Luxonis OAK-D runs stereo matching on its own vision processor; the navigation and hazard cameras of Mars rovers and some automotive forward systems (e.g. Subaru EyeSight) are stereo pairs. Software: OpenCV's stereoRectify, StereoBM, StereoSGBM and reprojectImageTo3D are ready-made versions of this whole lesson's pipeline; ROS's stereo_image_proc plugs them into a robot; on GPUs there are libSGM, NVIDIA VPI and learned stereo-disparity networks in Isaac ROS; the point clouds then go to Open3D / PCL (Lesson 12.1). Uses: forward stereo for drone obstacle avoidance, point clouds from a humanoid's head cameras for grasp planning (the grasp planning of Lesson 9.3), local costmaps of mobile robots; in visual odometry and SLAM a stereo pair gives metric scale directly (Part IV returns to this). Depth completion fuses sparse but reliable range (LiDAR, ToF) with dense images to fill the areas an active sensor misses.

    7常见误区Common pitfalls

    误区 1：视差的符号、方向和单位Pitfall 1: disparity sign, direction and units

    本节取左图为参考，\(d=u_L-u_R\ge0\)。有的库以右图为参考，有的把左右图交换，有的（OpenCV 的投影矩阵 \(P_2\)）把基线写成 \(T_x=-fB\) 这种带符号的量。结果是 \(Z\) 变成负数，或者"远处的东西比近处的东西近"。另一个经典陷阱：OpenCV 的 StereoBM / StereoSGBM 输出是定点数，要除以 16，无效像素是一个负的特殊值。直接把原始输出当像素视差代进 \(Z=fB/d\)，得到的深度只有真值的 1/16。永远用已知距离的物体做一次端到端的核对。

    This lesson uses the left image as reference, \(d=u_L-u_R\ge0\). Some libraries use the right image as reference, some swap the images, and some (OpenCV's projection matrix \(P_2\)) store the baseline as a signed quantity like \(T_x=-fB\). The symptom is negative \(Z\), or "far things nearer than near things". Another classic trap: the output of OpenCV's StereoBM / StereoSGBM is fixed-point and must be divided by 16, and invalid pixels carry a negative sentinel value. Feeding the raw output into \(Z=fB/d\) gives depths 16 times too small (1/16 of the truth). Always check end to end with an object at a known distance.

    误区 2：没矫正，或者矫正差一个像素Pitfall 2: not rectified, or rectified one pixel off

    块匹配假设同名点在同一行。直接对没有矫正的图像做 StereoBM 不会报错，只是悄悄变差。标定（10.4 节）的残余误差、镜头热胀冷缩、相机支架被撞歪，都会引入纵向错位。FIG 2 的"垂直错位"滑块说明了后果：对细纹理，1 个像素就足以让匹配从"基本正确"变成"几乎全错"；纹理越平滑越能容忍，但精度同样受损。出厂标定好的双目相机也要警惕：机器人摔过一次之后，就该重新检查对极线残差。

    Block matching assumes corresponding points are in the same row. Running StereoBM on unrectified images raises no error; it just quietly gets worse. Residual calibration error (Lesson 10.4), thermal drift of the lens, a knocked camera mount all introduce vertical misalignment. FIG 2's "vertical offset" slider shows the consequence: for fine texture a single pixel turns matching from "mostly right" to "almost all wrong"; smoother images tolerate more, but accuracy suffers just the same. Factory-calibrated stereo cameras deserve suspicion too: after a robot has taken a fall, re-check the epipolar residuals.

    误区 3：以为各个距离上的精度一样Pitfall 3: assuming the same precision at every range

    "这台相机深度精度 1%"只在某一个距离成立。\(\sigma_Z/Z=Z\sigma_d/(fB)\)，距离翻倍则相对误差翻倍、绝对误差翻四倍。FIG 3 里近处几乎看不到的散布，在远处是一个拖长尾的偏斜分布，不是对称的高斯。两个实际后果：①不要对深度做平均，而要对视差（噪声近似高斯的量）做平均或滤波，再取倒数；②给下游（建图、避障）的不确定性必须随深度变化，4.5 节的传播公式给出每个点自己的协方差，用一个固定的 σ 会在远处过度自信。

    "This camera has 1 % depth accuracy" holds at one distance only. \(\sigma_Z/Z=Z\sigma_d/(fB)\): double the range and the relative error doubles, the absolute error quadruples. The spread that is barely visible near in FIG 3 becomes a long-tailed skewed distribution far away, not a symmetric Gaussian. Two practical consequences: ① do not average depths; average or filter the disparities (whose noise is closer to Gaussian) and invert afterwards; ② the uncertainty handed to downstream code (mapping, avoidance) must vary with depth: Lesson 4.5's propagation gives each point its own covariance, and a fixed σ is over-confident far away.

    误区 4：纹理缺失与重复纹理会产生"自信的错误"Pitfall 4: missing or repeated texture yields confident errors

    FIG 2 的墙：代价曲线是平的，WTA 随便挑一个视差；平滑项（SGM）会从邻居那里借一个值，视差图看上去干净、完整，却是编的。重复纹理（栅栏、百叶窗、地砖）有多个同样深的谷。反光表面显示的是镜中的世界，不在表面上；透明物体匹配到背后的东西。对策：使用置信度（谷的深度、最优与次优代价的比值——OpenCV 的 uniquenessRatio、左右检验），丢掉不可信的点而不是填满；改用主动照明；或者让学习式方法利用语义先验。"稠密"和"正确"是两回事。

    FIG 2's wall: the cost curve is flat and WTA picks an arbitrary disparity; the smoothness term (SGM) borrows a value from neighbours, so the disparity map looks clean and complete but is made up. Repeated texture (fences, blinds, floor tiles) has several equally deep valleys. A reflective surface shows the mirror's world, which is not on the surface; a transparent object matches whatever lies behind it. Remedies: use confidence (valley depth, the ratio of best to second-best cost, as with OpenCV's uniquenessRatio, the left-right check) and drop unreliable points rather than fill everything; switch to active illumination; or let a learned method use semantic priors. "Dense" and "correct" are different things.

    误区 5：遮挡和视差搜索范围Pitfall 5: occlusion and the disparity search range

    深度突变的一侧，左图里有一条像素在右图里被前景挡住了（FIG 2 里紫色的半遮挡区），它们没有匹配；WTA 仍然会给出一个值，而且往往是错的，所以要靠左右检验或遮挡检测丢掉，再用邻近的背景值填充。另一个静默的失败：物体近到 \(d>D\)（FIG 1 的红区），匹配器根本不会去搜索真正的位置，只会在范围内挑一个虚假的最小值，输出的是一个看似合理的错误深度，而不是"太近了"的警告。选 \(D\) 要对应机器人真正需要的最近距离 \(Z_{\min}=fB/D\)；基线越大，近距盲区越大，也更多遮挡，这是大基线换取远处精度的代价。

    On one side of a depth jump, a band of left pixels is hidden behind the foreground in the right image (the purple half-occluded zone in FIG 2) and has no match; WTA still returns a value, often wrong, so drop them with the left-right check or an occlusion detector and fill from nearby background. Another silent failure: an object so near that \(d>D\) (FIG 1's red zone) is never searched at its true position; the matcher picks a spurious minimum inside the range and outputs a plausible but wrong depth rather than a "too close" warning. Choose \(D\) from the nearest distance the robot actually needs, \(Z_{\min}=fB/D\); a larger baseline widens the near blind zone and adds occlusion, the price paid for better far-range accuracy.

    8小练习Check yourself

    1. 矫正后的双目相机：焦距 \(f=700\) px，基线 \(B=0.12\) m，某像素的视差 \(d=21\) px。它的深度是多少？1. A rectified stereo camera has focal length \(f=700\) px and baseline \(B=0.12\) m. A pixel has disparity \(d=21\) px. What is its depth?
    \(Z=fB/d=700\times0.12/21=84/21=4.0\) m。\(f\) 和 \(d\) 都是像素，单位互相抵消，只剩 \(B\) 的米。\(Z=fB/d=700\times0.12/21=84/21=4.0\) m. \(f\) and \(d\) are both in pixels, so the units cancel and only the metres of \(B\) remain.
    2. 同一台相机，视差误差 \(\sigma_d=0.3\) px。在上题的 \(Z=4\) m 处，深度的标准差大约是多少？2. Same camera, disparity error \(\sigma_d=0.3\) px. At the previous question's \(Z=4\) m, about how large is the standard deviation of depth?
    \(\sigma_Z\approx Z^2\sigma_d/(fB)=16\times0.3/84=0.057\) m \(\approx5.7\) cm。也可以用相对误差：\(\sigma_Z/Z=\sigma_d/d=0.3/21\approx1.4\%\)，乘 4 m 得 5.7 cm。\(\sigma_Z\approx Z^2\sigma_d/(fB)=16\times0.3/84=0.057\) m \(\approx5.7\) cm. Equivalently the relative error is \(\sigma_Z/Z=\sigma_d/d=0.3/21\approx1.4\%\), and 1.4 % of 4 m is 5.7 cm.
    3. 要在两倍远的距离上保持同样的深度误差（\(\sigma_d\) 与 \(f\) 不变），基线应当变成原来的多少倍？3. To keep the same depth error at twice the distance (\(\sigma_d\) and \(f\) unchanged), by what factor must the baseline grow?
    \(\sigma_Z\propto Z^2/B\)。\(Z\to2Z\) 使分子乘 4，所以 \(B\) 也要乘 4 才能保持不变。同理焦距乘 4 也行（但视场角会变窄）；这就是为什么远距离双目相机的基线往往很长。\(\sigma_Z\propto Z^2/B\). \(Z\to2Z\) multiplies the numerator by 4, so \(B\) must also grow ×4. Raising \(f\) by 4 would work too (at the price of a narrower field of view); this is why long-range stereo rigs have long baselines.
    4. 一幅"已经矫正好"的图像对，某个同名点在左图是 \(v_L=240\)，在右图是 \(v_R=243\)。最合理的解释是？4. In an image pair supposedly "already rectified", a matching point has \(v_L=240\) in the left image and \(v_R=243\) in the right. What is the most sensible explanation?
    视差是水平方向的位置差 \(u_L-u_R\)；矫正好的同名点 \(v\) 应当相等。3 像素的行差说明矫正没做好（外参、畸变或相机被碰过）。FIG 2 里 1 个像素的纵向错位就足以让细纹理匹配崩溃。Disparity is the horizontal position difference \(u_L-u_R\); for a correctly rectified pair matching points have equal \(v\). A 3 px row difference means rectification is off (extrinsics, distortion, or a knocked camera). In FIG 2 a single pixel of vertical misalignment is enough to wreck fine-texture matching.
    5. 一台立体相机 \(f=500\) px，\(B=0.10\) m，匹配器的视差搜索范围是 \(D=64\) px。它能测到的最近深度大约是？5. A stereo camera has \(f=500\) px, \(B=0.10\) m and a disparity search range of \(D=64\) px. About how near can it measure?
    \(Z_{\min}=fB/D=500\times0.10/64=0.78\) m。更近的物体视差超过 64，搜索不到（FIG 1 的红区）。要看到 0.4 m，需要 \(D\approx125\)，或者缩小基线，或者降低分辨率（\(f\) 以像素计，也随之减半）。\(Z_{\min}=fB/D=500\times0.10/64=0.78\) m. Anything nearer has disparity above 64 and is never searched (FIG 1's red zone). To see 0.4 m you need \(D\approx125\), a smaller baseline, or a lower resolution (\(f\), in pixels, then halves too).

    9延伸阅读Further reading

    • R. Hartley & A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. (Cambridge University Press, 2003), Ch. 11 — 基础矩阵的计算与图像矫正（射影矫正）Computing the fundamental matrix and image rectification (projective rectification)
    • D. Scharstein & R. Szeliski, A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, International Journal of Computer Vision 47(1–3), 2002 — 稠密立体算法的"匹配代价—聚合—优化—精化"四步框架与评测The cost / aggregation / optimisation / refinement framework for dense stereo, and its evaluation
    • H. Hirschmüller, Stereo processing by semiglobal matching and mutual information, IEEE TPAMI 30(2), 2008 — SGM 的原始论文The original SGM paper
    • R. Szeliski, Computer Vision: Algorithms and Applications, 2nd ed. (Springer, 2022), Ch. 12 — 深度估计：对极几何、稠密对应、全局优化与深度网络Depth estimation: epipolar geometry, dense correspondence, global optimisation and deep networks
    • L. Keselman, J. I. Woodfill, A. Grunnet-Jepsen & A. Bhowmik, Intel RealSense Stereoscopic Depth Cameras, CVPR Workshops, 2017 — 主动红外双目相机的设计与取舍The design and trade-offs of active IR stereo cameras
    • A. Fusiello, E. Trucco & A. Verri, A compact algorithm for rectification of stereo pairs, Machine Vision and Applications 12, 2000 — 本节矫正构造的出处The source of the rectification construction used here
    • A. Kendall et al., End-to-End Learning of Geometry and Context for Deep Stereo Regression (GC-Net), ICCV 2017; J.-R. Chang & Y.-S. Chen, Pyramid Stereo Matching Network (PSMNet), CVPR 2018; L. Lipson, Z. Teed & J. Deng, RAFT-Stereo, 3DV 2021 — 学习式立体：在特征代价体上做 3D 卷积，或迭代更新视差Learned stereo: 3D convolutions on a feature cost volume, or iterative disparity updates
    • B. Julesz, Foundations of Cyclopean Perception (University of Chicago Press, 1971) — 随机点立体图与人类立体视觉Random-dot stereograms and human stereopsis