Part III · 第 12 章 三维表示与配准 · 12.4Part III · Ch. 12 3D Representations & Registration · 12.4

NeRF 与 3D Gaussian SplattingNeRF & 3D Gaussian Splatting

照片是三维世界的二维投影。辐射场反过来做：把场景写成一个可以被渲染的函数（NeRF 的神经网络）或一团可以被渲染的高斯（3D Gaussian Splatting），渲染过程处处可微，于是让梯度下降去调场景参数，直到渲染出的图和已知位姿的照片一致。本节把体渲染、位置编码、协方差投影和泼溅混合都放进真正在页面里运行的优化实验。 A photograph is a 2-D projection of the 3-D world. Radiance fields run it backwards: write the scene as a renderable function (NeRF's neural network) or as a cloud of renderable Gaussians (3D Gaussian Splatting), make rendering differentiable everywhere, and let gradient descent adjust the scene until its renderings match photographs with known poses. This lesson puts volume rendering, positional encoding, covariance projection and splat blending into optimizations that really run in the page.

约 40 分钟~40 min 前置：2.3 积分 · 3.3 雅可比 · 4.5 不确定性传播 · 6.3 梯度下降 · 10.3 相机 · 11.6 束调整 · 12.3 渲染管线Prerequisites: 2.3 integration · 3.3 Jacobian · 4.5 uncertainty propagation · 6.3 gradient descent · 10.3 cameras · 11.6 bundle adjustment · 12.3 graphics pipeline 5 个交互5 interactives

    1为什么机器人需要它Why robots need this

    第 11 章的 SfM 与 MVS 给出稀疏点云或网格，拿来定位、避障已经够用，但有两个老毛病：玻璃、反光面、细线、树叶和头发几乎重建不出来；贴上纹理的网格看上去也不像照片，因为外观会随视角变（高光、折射、半透明）。辐射场（radiance field）换了一个问法：不去求"表面在哪"，只问"站在这里、朝这个方向看，会看到什么颜色"。

    关键在于让渲染本身可微。如果从场景参数到像素颜色的每一步都有导数，就能把"渲染图与照片的差"当成损失，对场景参数做梯度下降（6.3、6.4 节），其中相机位姿由 SfM / 束调整（11.6 节）事先给出。2020 年的 NeRF 用一个小神经网络当场景，2023 年的 3D Gaussian Splatting（3DGS）用几十万到几百万个半透明的三维高斯椭球当场景。机器人用它们做：

    • 场景重建与 real2sim：手机绕物体拍一圈，得到可以真实感渲染、可以导入仿真的资产和数字孪生。
    • 杂乱场景里的操作：透明、反光物体的几何；把抓取规划放进辐射场里做（Dex-NeRF、GraspNeRF 等，见第 6 节）。
    • 建图与 SLAM：地图本身可渲染，相机位姿可以靠"渲染误差最小"来跟踪（iMAP、NICE-SLAM、SplaTAM 等）。
    • 数据与世界模型：新视角合成做数据增强；它也是"机器人脑中的世界长什么样"的一个候选表示（第七部分）。

    本节沿着这条线走：先看渲染一个像素到底在算什么（FIG 1），再用"泼溅 + 混合"在 2D 里亲手把一张图拟合出来（FIG 2，核心实验），然后看 NeRF 为什么需要位置编码（FIG 3），以及三维椭球怎样变成屏幕上的二维椭圆（FIG 4），最后比较两者的开销（FIG 5）。

    The SfM and MVS of Chapter 11 give a sparse point cloud or a mesh, good enough to localize and avoid obstacles, but with two old weaknesses: glass, shiny surfaces, thin wires, foliage and hair barely reconstruct, and a textured mesh does not look like a photograph because appearance changes with the viewpoint (highlights, refraction, translucency). A radiance field asks a different question: instead of "where is the surface", it asks "standing here and looking in this direction, what colour do I see?"

    The key is to make rendering itself differentiable. If every step from scene parameters to pixel colour has a derivative, the difference between the rendering and the photograph is a loss and the scene parameters can be fitted by gradient descent (Lessons 6.3, 6.4), with the camera poses supplied beforehand by SfM / bundle adjustment (Lesson 11.6). NeRF (2020) uses a small neural network as the scene; 3D Gaussian Splatting (3DGS, 2023) uses hundreds of thousands to millions of translucent 3-D Gaussian ellipsoids. Robots use them for:

    • Scene reconstruction and real2sim: walk around an object with a phone and get an asset or digital twin that renders photo-realistically and can be imported into simulation.
    • Manipulation in clutter: the geometry of transparent and reflective objects; grasp planning inside a radiance field (Dex-NeRF, GraspNeRF and others, Section 6).
    • Mapping and SLAM: the map itself is renderable and the camera pose can be tracked by minimizing the rendering error (iMAP, NICE-SLAM, SplaTAM, ...).
    • Data and world models: novel-view synthesis for data augmentation; also a candidate for "what the world looks like inside the robot's head" (Part VII).

    The lesson follows that line: first what rendering one pixel actually computes (FIG 1), then a hands-on 2-D "splat and blend" fit of an image (FIG 2, the core lab), then why NeRF needs positional encoding (FIG 3) and how a 3-D ellipsoid becomes a 2-D ellipse on screen (FIG 4), and finally the cost comparison (FIG 5).

    2直觉：一个像素 = 沿光线对"雾"的积分Intuition: one pixel = an integral through "fog" along a ray

    把场景想成一团带颜色的雾。一条相机光线穿过雾：每走一小段 \(\mathrm{d}t\)，它有 \(\sigma(t)\,\mathrm{d}t\) 的概率"撞上"什么并带着那里的颜色停下来，\(\sigma\) 叫体密度（单位 1/米）。走到 \(t\) 还没被拦下的概率是透射率 \(T(t)=\exp\!\big(-\!\int_0^t\sigma\big)\)；于是"恰好在 \(t\) 处停下"的概率密度是 \(T(t)\sigma(t)\)，像素颜色就是停下位置颜色的期望：

    Think of the scene as coloured fog. A camera ray passes through it: over each small step \(\mathrm{d}t\) it has probability \(\sigma(t)\,\mathrm{d}t\) of "hitting something" and stopping with that place's colour; \(\sigma\) is the volume density (units 1/m). The probability of surviving to \(t\) is the transmittance \(T(t)=\exp\!\big(-\!\int_0^t\sigma\big)\), so the density of "stopping exactly at \(t\)" is \(T(t)\sigma(t)\), and the pixel colour is the expected colour of the stopping point:

    $$ C(\mathbf r)=\int_{t_n}^{t_f}\hl{T}{T(t)}\,\sigma(\mathbf r(t))\,\mathbf c(\mathbf r(t),\mathbf d)\,\mathrm{d}t,\qquad T(t)=\exp\!\Big(-\!\int_{t_n}^{t}\sigma(\mathbf r(s))\,\mathrm{d}s\Big). $$

    实心墙是一层 \(\sigma\) 极大的薄壳（光线几乎必然在那里停下），雾是很小的 \(\sigma\) 铺一大段（每处只拦下一点点）。计算机里把光线切成 \(N\) 段、每段宽 \(\delta_i\)：这一段拦下光线的概率是 \(\alpha_i=1-e^{-\sigma_i\delta_i}\)，之前一路没被拦下的概率是 \(T_i=\prod_{j<i}(1-\alpha_j)\)，于是权重 \(w_i=T_i\alpha_i\)，\(\hat C=\sum_i w_i\mathbf c_i\)。这正是 12.3 节里从前往后的 alpha 混合。下图的密度画在 16 个小格上（拖动条形顶端的圆圈，或直接点击某一格，改写自己的密度）：

    A solid wall is a thin shell of huge \(\sigma\) (the ray almost surely stops there); fog is a small \(\sigma\) spread over a long stretch (each place stops only a little). On a computer the ray is cut into \(N\) intervals of width \(\delta_i\): the probability that interval \(i\) stops the ray is \(\alpha_i=1-e^{-\sigma_i\delta_i}\), the probability of getting through everything before it is \(T_i=\prod_{j<i}(1-\alpha_j)\), so the weight is \(w_i=T_i\alpha_i\) and \(\hat C=\sum_i w_i\mathbf c_i\). This is exactly the front-to-back alpha blending of Lesson 12.3. The density below lives on 16 small cells (drag the circles on top of the bars, or click a cell, to write your own density):

    FIG 1沿一条光线的体渲染Volume rendering along one ray
    σ(t)（拖圆圈）与 T(t)σ(t) (drag circles) and T(t)
    权重密度 Tσ 与采样权重 wweight density Tσ and sample weights w
    预设Presets
    左：条形是 16 格分段常数的密度 \(\sigma\)（颜色是该格材质的颜色，近红远蓝），黄线是精确的透射率 \(T(t)\)，底部短竖线是 \(N\) 个采样点。右：彩色条是 \(N\) 段的权重 \(w_i/\delta\)（面积 = 权重），紫线是精确的权重密度 \(T\sigma\)，竖线是期望深度（实线：采样求和；虚线：精确）。"Max" 用 \(\alpha=1-e^{-\sigma\delta}\)，"朴素"用 \(\alpha=\sigma\delta\)（直接做黎曼和）。"前 k 个"滑块只合成前 \(k\) 个采样，就是前向 alpha 混合的中间状态。 Left: the bars are the 16-cell piecewise-constant density \(\sigma\) (colour = that cell's material colour, red near to blue far), the amber line is the exact transmittance \(T(t)\), the short ticks at the bottom are the \(N\) samples. Right: coloured bars are the \(N\) weights \(w_i/\delta\) (area = weight), the purple line is the exact weight density \(T\sigma\), vertical lines are the expected depth (solid: from the samples; dashed: exact). "Max" uses \(\alpha=1-e^{-\sigma\delta}\), "naive" uses \(\alpha=\sigma\delta\) (a plain Riemann sum). The "first k" slider composites only the first \(k\) samples, an intermediate state of front-to-back blending.
    试一试Try this
    • 点"实心墙"：几乎所有权重压在墙的第一个格子上，\(\sum w\approx1\)，期望深度就是墙的位置。墙后面的东西为什么对像素毫无贡献？Press "solid wall": nearly all the weight piles onto the wall's first cell, \(\sum w\approx1\), and the expected depth is the wall's position. Why does anything behind the wall not contribute to the pixel?
    • 把 \(N\) 减到 16 以下（比如 5）：采样点可能"跨过"薄墙，颜色误差立刻出现。窄的东西需要足够多的采样。Drop \(N\) below 16 (say 5): the samples can step over a thin wall and a colour error appears at once. Narrow things need enough samples.
    • 再把 \(N\) 调成 16 的倍数（16、32、48……）：误差降到 \(10^{-9}\) 以下。为什么这时求和和积分完全相等？（密度在每格内是常数。）Now set \(N\) to a multiple of 16 (16, 32, 48, ...): the error falls below \(10^{-9}\). Why are the sum and the integral then exactly equal? (The density is constant inside each cell.)
    • 切到"朴素"并选"实心墙"：\(\sum w\) 超过 1，颜色被"过曝"。\(\alpha=1-e^{-\sigma\delta}\) 保证了什么？Switch to "naive" with the wall: \(\sum w\) exceeds 1 and the colour is over-exposed. What does \(\alpha=1-e^{-\sigma\delta}\) guarantee?
    核心洞见Key insight

    体渲染把"看到什么"写成了一个对密度和颜色都光滑可微的公式：\(\hat C=\sum_i T_i\alpha_i\mathbf c_i\) 只含乘法、指数和求和。所以只要有任何能产生 \(\sigma_i,\mathbf c_i\) 的参数化（神经网络、体素格、一堆高斯），损失 \(\|\hat C-C_{\text{照片}}\|^2\) 对这些参数的梯度都存在，可以直接梯度下降。NeRF 和 3DGS 的差别，只在于"\(\sigma_i,\mathbf c_i\) 从哪里来"。

    Volume rendering writes "what is seen" as a formula that is smooth and differentiable in both density and colour: \(\hat C=\sum_i T_i\alpha_i\mathbf c_i\) contains only multiplications, exponentials and sums. So for any parametrization that produces \(\sigma_i,\mathbf c_i\) (a neural network, a voxel grid, a bag of Gaussians), the loss \(\|\hat C-C_{\text{photo}}\|^2\) has a gradient with respect to its parameters and plain gradient descent applies. NeRF and 3DGS differ only in "where \(\sigma_i,\mathbf c_i\) come from".

    3动手玩：用一把高斯"泼"出一张图Play: splat an image out of a handful of Gaussians

    3DGS 的做法是把场景直接写成许多半透明、有朝向的椭球。每个椭球有位置 \(\boldsymbol\mu\)、形状 \(\Sigma\)、不透明度 \(o\)、颜色 \(\mathbf c\)。渲染时把它们投影到屏幕成二维椭圆（"泼溅"，splat），按深度从前到后排序，在每个像素上做 alpha 混合。下面的实验把相机拿掉，只留最后一步：屏幕上有 \(N\) 个二维高斯，每个有位置、两个尺度 \(s_1,s_2\)、朝向 \(\theta\)、颜色、不透明度和一个固定的深度 \(z\)；像素 \(\mathbf x\) 处的颜色是

    3DGS writes the scene directly as many translucent, oriented ellipsoids. Each has a position \(\boldsymbol\mu\), a shape \(\Sigma\), an opacity \(o\) and a colour \(\mathbf c\). To render, project them to 2-D ellipses on screen ("splatting"), sort by depth front to back, and alpha-blend at each pixel. The lab below drops the camera and keeps only that last step: \(N\) 2-D Gaussians on screen, each with a position, two scales \(s_1,s_2\), an orientation \(\theta\), a colour, an opacity and a fixed depth \(z\); the colour at pixel \(\mathbf x\) is

    $$ C(\mathbf x)=\sum_{i}\mathbf c_i\,\hl{alpha}{\alpha_i}\prod_{j<i}(1-\alpha_j),\qquad \alpha_i=o_i\exp\!\Big(-\tfrac12(\mathbf x-\boldsymbol\mu_i)^{\!\top}\Sigma_i^{-1}(\mathbf x-\boldsymbol\mu_i)\Big),\quad \Sigma_i=R(\theta_i)\,\mathrm{diag}(s_{1}^2,s_{2}^2)\,R(\theta_i)^{\!\top}, $$

    其中下标已按深度从前到后排好。这是真正的优化：损失是渲染图与目标图的均方误差，对每个高斯的全部 9 个参数（位置、两个对数尺度、角度、颜色、不透明度）的梯度是手写的解析梯度（页面里已和有限差分核对过），用 Adam 更新。"稠密化"和"剪枝"复刻了 3DGS 的自适应密度控制：位置梯度长期很大的高斯说明"这里还没拟合好"，大的被分裂（这里把两个子高斯沿长轴摆开，原版是从母高斯的分布里采样）、小的被克隆；贡献几乎为零的被剪掉。点击图中的高斯可以选中它，然后拖动三个圆圈来改位置、长轴、短轴，亲手感受这种表示。

    with the index already sorted front to back. This is a real optimization: the loss is the mean squared error between the render and the target, the gradient with respect to all 9 parameters of every Gaussian (position, two log-scales, angle, colour, opacity) is a hand-written analytic gradient (checked against finite differences), and Adam does the update. "Densify" and "Prune" mimic 3DGS's adaptive density control: a Gaussian whose positional gradient stays large says "this region is not fitted yet", so big ones are split (here the two children are placed along the long axis, the original samples them from the parent's distribution) and small ones cloned; Gaussians that contribute almost nothing are pruned. Click a Gaussian to select it, then drag the three circles to move it or change its long and short axes, and feel the representation by hand.

    FIG 2泼溅与混合：二维高斯拟合实验室Splat and blend: a 2-D Gaussian fitting lab
    渲染结果 64×48（点击选中高斯）RENDER 64×48 · click a Gaussian
    左：当前渲染（"椭圆"视图把每个高斯画成 2σ 的实心椭圆，"两者"在混合图上叠加轮廓）。右上：目标图与误差图（越亮误差越大）；右下：PSNR 随迭代次数的曲线，竖虚线标出稠密化 / 剪枝的时刻。选中高斯后出现三个圆圈：中心（位置）、长轴端点（朝向与长轴）、短轴端点（短轴）。N 滑块会重新初始化。 Left: the current render (the "ellipses" view draws each Gaussian as a solid 2σ ellipse; "both" overlays outlines on the blended image). Top right: the target and the error map (brighter = larger error); bottom right: PSNR against iteration, with dashed verticals at densify / prune events. A selected Gaussian shows three circles: centre (position), end of the first axis (orientation and first scale), end of the second axis (second scale). The N slider re-initializes.
    试一试Try this
    • 点"拟合"，看渲染从色块逐渐变成目标图，PSNR 升到 24 dB 以上。前几十步进步最快：先抓大块颜色，再修细节。Press "Fit" and watch the render turn from colour blobs into the target, with PSNR rising past 24 dB. The first dozens of steps gain the most: big colour areas first, details later.
    • 选中一个高斯，把它拖离原位 3 个像素以上，PSNR 立刻下降；再点"拟合"，优化器把它拉回去（或者让别的高斯接手）。Select a Gaussian and drag it more than 3 pixels away: PSNR drops at once; press "Fit" and the optimizer pulls it back (or lets its neighbours take over).
    • 选"棋盘"目标，N 设成 20，拟合一阵后点"稠密化"几次：高斯数增加，PSNR 继续上升。哪些地方的高斯最先被分裂？（锐利的边。）Choose the "checker" target, set N to 20, fit for a while and press "Densify" a few times: the Gaussian count grows and PSNR keeps rising. Where do the splits happen first? (At sharp edges.)
    同一个公式，两种"泼法"One formula, two ways to fill it in

    把 FIG 1 的 \(\hat C=\sum T_i\alpha_i\mathbf c_i\) 和这里的 \(C=\sum\mathbf c_i\alpha_i\prod_{j<i}(1-\alpha_j)\) 并排看：形式一模一样，区别是 \(\alpha_i\) 怎么来。NeRF 沿光线采样空间点，\(\alpha_i\) 由网络在该点的密度给出；3DGS 沿光线遇到高斯，\(\alpha_i\) 由这个高斯投影到像素处的值给出。前者每个像素要问网络几百次，后者只需要对覆盖这个像素的少数高斯做排序混合。

    Put FIG 1's \(\hat C=\sum T_i\alpha_i\mathbf c_i\) next to this lab's \(C=\sum\mathbf c_i\alpha_i\prod_{j<i}(1-\alpha_j)\): the form is identical, and what differs is where \(\alpha_i\) comes from. NeRF samples points in space along the ray and gets \(\alpha_i\) from the network's density there; 3DGS meets Gaussians along the ray and gets \(\alpha_i\) from the value of each Gaussian's projection at the pixel. The former queries the network hundreds of times per pixel; the latter sorts and blends only the few Gaussians covering the pixel.

    4第二、第三视角：位置编码、椭球的投影与两者的比较Second and third views: positional encoding, ellipsoid projection, and the comparison

    NeRF 的另一半：为什么直接喂坐标，网络画不出锐利的边NeRF's other half: why a network fed raw coordinates cannot draw sharp edges

    NeRF 的场景是一个多层感知机 \(F_\Theta:(\mathbf x,\mathbf d)\mapsto(\sigma,\mathbf c)\)。如果直接把坐标喂给网络，它倾向于先学会低频的平滑成分，再极其缓慢地学高频，这叫谱偏置（spectral bias，Rahaman 等 2019）。NeRF 的办法是先把坐标做傅里叶编码 \(\gamma(x)=(\sin 2^k\pi x,\cos 2^k\pi x)_{k=0}^{L-1}\)，再喂给网络（Tancik 等 2020 从神经正切核的角度解释了为什么这样有效）。下面用一维的"场景"把这个现象缩小到可以现场训练：目标是一条带锐利边的曲线，两个网络（2 个隐藏层、各 32 个单元，约 1.2 到 1.6 千个参数）用同样的 Adam 同时训练，一个吃原始的 \(x\)，一个吃 \(\gamma(x)\)。反向传播会在第五部分讲，这里把训练循环当黑盒，里面的数学（手写的反向传播）已经用有限差分验证过。

    NeRF's scene is a multilayer perceptron \(F_\Theta:(\mathbf x,\mathbf d)\mapsto(\sigma,\mathbf c)\). Fed raw coordinates, a network tends to learn the smooth low-frequency components first and the high frequencies only extremely slowly, the spectral bias (Rahaman et al. 2019). NeRF's remedy is to Fourier-encode the coordinate, \(\gamma(x)=(\sin 2^k\pi x,\cos 2^k\pi x)_{k=0}^{L-1}\), before feeding the network (Tancik et al. 2020 explain why this works through the neural tangent kernel). The experiment below shrinks the effect into a 1-D "scene" that trains in front of you: the target is a curve with sharp edges, and two networks (2 hidden layers of 32 units, about 1.2 to 1.6 thousand parameters) are trained simultaneously with the same Adam, one on the raw \(x\) and one on \(\gamma(x)\). Backpropagation is taught in Part V, so treat the training loop as a black box here; its maths (a hand-written backward pass) has been verified against finite differences.

    FIG 3位置编码：NeRF 的缩微版Positional encoding: NeRF in miniature
    灰线是目标，灰点是 128 个训练样本（只在这些点上算损失）。紫色：只吃原始 \(x\) 的网络；黄色：吃 \(\gamma(x)\) 的网络。右图实线是在 400 个没训练过的点上的测试损失，虚线是训练损失。把 \(L\) 调大，黄色网络在训练点上拟合得越来越好，但样本之间会出现噪声：测试损失先降后升。 The grey line is the target, grey dots are the 128 training samples (the only places the loss sees). Purple: the network fed the raw \(x\); amber: the network fed \(\gamma(x)\). On the right, solid lines are the test loss on 400 unseen points and dashed lines the training loss. Raise \(L\) and the amber network fits the training points ever better, but noise appears between the samples: the test loss falls and then rises.
    试一试Try this
    • 用"条纹"目标、\(L=4\) 训练 600 步以上：紫色网络仍是一条平缓的波，测试损失停在 \(10^{-1}\) 量级；黄色网络已经画出清晰的方波。With the "stripes" target and \(L=4\), train for 600+ steps: the purple network is still a gentle wave with test loss around \(10^{-1}\), while the amber one already draws a crisp square wave.
    • 把 \(L\) 增到 8 以上再训练：训练损失趋近 0，但样本之间出现高频噪声，测试损失比训练损失高几个数量级。编码频率要和数据密度匹配。Raise \(L\) to 8 or more and train: the training loss goes to nearly 0 but high-frequency noise appears between samples, and the test loss is orders of magnitude above the training loss. The encoding bandwidth has to match the data density.

    3DGS 的几何：三维椭球怎样变成二维椭圆3DGS geometry: how a 3-D ellipsoid becomes a 2-D ellipse

    FIG 2 里二维椭圆是自由参数；真实的 3DGS 里它们是三维高斯投影的结果。一个三维高斯 \(\mathcal N(\boldsymbol\mu,\Sigma)\) 的协方差写成 \(\Sigma=R\,S\,S^{\top}R^{\top}\)（\(S\) = 三个尺度的对角阵，\(R\) = 旋转，1.11、1.12 节：特征向量是 \(R\) 的列，特征值是尺度的平方）。相机把它投到图像上时，透视投影是非线性的，但在 \(\boldsymbol\mu\) 附近可以用一阶泰勒展开（3.3 节的雅可比）近似成仿射映射，而高斯经过仿射映射仍是高斯，协方差按 4.5 节的传播公式变换：\(\Sigma'=J\,W\,\Sigma\,W^{\top}J^{\top}\)（\(J\) 是 \(2\times3\) 的，结果直接就是 \(2\times2\)）。这与 EKF 里把非线性观测线性化是同一个技巧（4.8 节），在图形学里叫 EWA 泼溅（Zwicker 等，2001）。

    下图用蒙特卡洛检验它：从三维高斯采样 20000 个点，逐点做精确的透视投影，算出它们的经验协方差，与公式的 \(\Sigma'\) 比较。椭球小、离相机远时二者几乎重合（剩余的 1% 左右是采样噪声）；椭球大、离相机近时，雅可比在椭球范围内变化显著，一阶近似就失效了。

    In FIG 2 the 2-D ellipses were free parameters; in real 3DGS they are the result of projecting 3-D Gaussians. A 3-D Gaussian \(\mathcal N(\boldsymbol\mu,\Sigma)\) has the covariance \(\Sigma=R\,S\,S^{\top}R^{\top}\) (\(S\) = diagonal matrix of three scales, \(R\) = a rotation; Lessons 1.11, 1.12: the eigenvectors are the columns of \(R\) and the eigenvalues are the squared scales). Perspective projection is nonlinear, but near \(\boldsymbol\mu\) it can be approximated by a first-order Taylor expansion (the Jacobian of Lesson 3.3) as an affine map; a Gaussian pushed through an affine map is still Gaussian and its covariance transforms by the propagation rule of Lesson 4.5: \(\Sigma'=J\,W\,\Sigma\,W^{\top}J^{\top}\) (with the \(2\times3\) Jacobian \(J\) the result is directly \(2\times2\)). It is the same trick as linearizing a nonlinear measurement in the EKF (Lesson 4.8), and graphics calls it EWA splatting (Zwicker et al., 2001).

    The figure tests it by Monte Carlo: draw 20000 points from the 3-D Gaussian, project each one with the exact perspective map, compute their empirical covariance and compare with the formula's \(\Sigma'\). For a small ellipsoid far from the camera the two coincide (the remaining ~1% is sampling noise); for a large ellipsoid close to the camera the Jacobian changes substantially across the ellipsoid and the first-order approximation breaks down.

    FIG 43D 高斯的投影：J W Σ Wᵀ JᵀProjecting a 3-D Gaussian: J W Σ Wᵀ Jᵀ
    世界：椭球 (2σ) 与相机，拖动旋转WORLD · 2σ ellipsoid & camera · drag to orbit
    左：黄色线框是椭球的 2σ 曲面，三根彩色轴是 \(R\) 的列乘 \(2s_i\)，青色是相机视锥，黄点是几个采样点。右：黄点是 20000 个采样点中前 700 个的精确投影；绿色实线椭圆是公式 \(\Sigma'\) 的 2σ 等值线，紫色虚线是经验协方差的 2σ 等值线。读数里的相对误差是 \(\|\Sigma'-\hat\Sigma_{MC}\|_F/\|\hat\Sigma_{MC}\|_F\)。 Left: the amber wireframe is the ellipsoid's 2σ surface, the three coloured axes are the columns of \(R\) times \(2s_i\), cyan is the camera frustum, and a few sample points are drawn. Right: amber dots are the exact projections of the first 700 of 20000 samples; the solid green ellipse is the 2σ contour of the formula's \(\Sigma'\), the dashed purple one the 2σ contour of the empirical covariance. The relative error in the readout is \(\|\Sigma'-\hat\Sigma_{MC}\|_F/\|\hat\Sigma_{MC}\|_F\).
    试一试Try this
    • 保持椭球较小、相机较远（\(s_{\max}/z<0.1\)）：绿色实线和紫色虚线几乎重合，相对误差在 3% 以内。Keep the ellipsoid small and the camera far (\(s_{\max}/z<0.1\)): the solid green and dashed purple ellipses nearly coincide, relative error within 3%.
    • 把椭球放大、相机拉近到 3 以内：相对误差超过 10%，两个椭圆明显错开。一阶近似的适用范围就是"椭球尺寸远小于它到相机的距离"。Enlarge the ellipsoid and pull the camera inside distance 3: the relative error exceeds 10% and the ellipses visibly part. The first-order approximation is valid when "the ellipsoid is much smaller than its distance to the camera".

    NeRF 与 3DGS：诚实的对比NeRF vs 3DGS: an honest comparison

    下表是一般性的、会随具体实现变化的概括，不是基准测试；"量级"只是数量级。

    The table gives general statements that vary with the implementation; it is not a benchmark, and the "orders of magnitude" are just that.

    NeRF（隐式）NeRF (implicit) 3D Gaussian Splatting（显式）3D Gaussian Splatting (explicit)
    表示RepresentationMLP（常配哈希网格等特征网格）：\(F_\Theta(\mathbf x,\mathbf d)\to(\sigma,\mathbf c)\)an MLP (often with a hash grid or feature grid): \(F_\Theta(\mathbf x,\mathbf d)\to(\sigma,\mathbf c)\)几十万到几百万个三维高斯，每个约 59 个浮点数（含 3 阶球谐）105 to 106+ 3-D Gaussians, about 59 floats each (including degree-3 spherical harmonics)
    渲染Rendering光线步进，每像素几十到几百次网络 / 特征查询；原版远不到实时，加速版本（Instant-NGP 等）可交互ray marching, tens to hundreds of network or feature queries per pixel; the original is far from real time, accelerated variants (Instant-NGP, ...) are interactive排序后的光栅化（按 16×16 分块），常见报告是桌面 GPU 上 1080p 实时sorted rasterization (16×16 tiles); real time at 1080p on a desktop GPU is commonly reported
    训练Training原版数小时到一天；哈希网格版本几秒到几分钟original: hours to a day; hash-grid variants: seconds to minutes一个场景约几分钟到半小时（取决于 GPU 与场景）minutes to about half an hour per scene (depending on GPU and scene)
    内存Memory小（几 MB 到几十 MB 的权重 / 特征网格）small (MBs to tens of MBs of weights / feature grids)大：100 万高斯 ≈ 59×4 B×106 ≈ 240 MB，大场景可到 GBlarge: 106 Gaussians ≈ 59×4 B×106 ≈ 240 MB, GBs for large scenes
    可编辑性Editability难：知识藏在权重里hard: the knowledge is buried in weights相对容易：删除、移动、合并高斯；但没有语义分组comparatively easy: delete, move, merge Gaussians, though there is no semantic grouping
    几何Geometry密度场，没有明确表面；SDF 变体（NeuS、VolSDF）给出表面（12.2 节）a density field with no explicit surface; SDF variants (NeuS, VolSDF) give surfaces (Lesson 12.2)一团半透明椭球，不是水密表面；2DGS 等变体改善几何a cloud of translucent ellipsoids, not a watertight surface; variants such as 2DGS improve the geometry
    画质Quality好；抗锯齿的变体（Mip-NeRF 360、Zip-NeRF）在无界场景最强之一good; anti-aliased variants (Mip-NeRF 360, Zip-NeRF) are among the best on unbounded scenes相当或更好，视场景而定；稀疏视角、远离训练视角时都会出现伪影comparable or better depending on the scene; both show artifacts under sparse views and far from the training views

    5再看数学The math

    提示：把鼠标悬停在公式中带底色的符号上，FIG 1、FIG 2 或 FIG 4 中对应的元素会被高亮。Tip: hover the highlighted symbols in the formulas; the matching element in FIG 1, FIG 2 or FIG 4 lights up.

    体渲染的求积：为什么是 \(1-e^{-\sigma\delta}\)，为什么可微Quadrature of volume rendering: why \(1-e^{-\sigma\delta}\), and why it is differentiable

    设光线被切成 \(N\) 段，第 \(i\) 段内密度 \(\sigma_i\)、颜色 \(\mathbf c_i\) 为常数，宽 \(\delta_i\)。段内 \(T(t)=T_i\,e^{-\sigma_i(t-t_i)}\)，所以这一段对积分的贡献可以直接积出来：

    Cut the ray into \(N\) intervals with density \(\sigma_i\) and colour \(\mathbf c_i\) constant on interval \(i\) of width \(\delta_i\). Inside the interval \(T(t)=T_i\,e^{-\sigma_i(t-t_i)}\), so its contribution to the integral can be done in closed form:

    $$ \begin{aligned}\int_{t_i}^{t_i+\delta_i}\!T(t)\,\sigma_i\,\mathbf c_i\,\mathrm{d}t&=\mathbf c_i\,T_i\!\int_0^{\delta_i}\!\sigma_ie^{-\sigma_is}\mathrm{d}s=\mathbf c_i\,T_i\,\underbrace{\big(1-e^{-\sigma_i\delta_i}\big)}_{\alpha_i},\\ \hat C&=\sum_{i=1}^{N}\hl{w}{w_i}\,\mathbf c_i,\quad w_i=T_i\alpha_i,\quad T_i=\prod_{j<i}(1-\alpha_j)=e^{-\sum_{j<i}\sigma_j\delta_j}.\end{aligned} $$

    这是 Max（1995）的结果，也是 NeRF 的式 (3)。当 \(\sigma,\mathbf c\) 在每段内真是常数时它精确等于积分（FIG 1 里 \(N\) 为 16 的倍数时误差 \(<10^{-9}\)，已在 node 中核对）。因为 \(\prod(1-\alpha_j)\) 是"一路没被拦下"的概率，所以 \(\sum_iw_i=1-T_{N+1}\le1\)：剩下的 \(T_{N+1}\) 分给背景。朴素黎曼和 \(\alpha=\sigma\delta\) 没有这个保证。实现上从前往后累加 \(C\leftarrow C+T\alpha\mathbf c,\ T\leftarrow T(1-\alpha)\)，\(T\) 低于 \(10^{-4}\) 左右就可以提前终止，和 12.3 节的 alpha 混合一致。

    This is Max's result (1995) and Eq. (3) of the NeRF paper. When \(\sigma,\mathbf c\) really are constant on each interval it equals the integral exactly (in FIG 1 the error is \(<10^{-9}\) when \(N\) is a multiple of 16, checked in node). Because \(\prod(1-\alpha_j)\) is the probability of getting through everything before, \(\sum_iw_i=1-T_{N+1}\le1\): the remaining \(T_{N+1}\) goes to the background. The naive Riemann sum \(\alpha=\sigma\delta\) has no such guarantee. In code one accumulates front to back, \(C\leftarrow C+T\alpha\mathbf c,\ T\leftarrow T(1-\alpha)\), and may stop early once \(T\) falls below about \(10^{-4}\), the same as the alpha blending of Lesson 12.3.

    梯度。对每个采样有 \(\partial\hat C/\partial\mathbf c_k=w_k\)，\(\partial\hat C/\partial\alpha_k=T_k\,(\mathbf c_k-\mathbf C_{>k})\)，其中 \(\mathbf C_{>k}\) 是"第 \(k\) 个之后所有东西"混合出的颜色（从后往前递推 \(\mathbf C_{>k-1}=\alpha_k\mathbf c_k+(1-\alpha_k)\mathbf C_{>k}\)，初值为背景）。直觉很清楚：提高 \(\alpha_k\) 就是把后面的颜色换成 \(\mathbf c_k\)，影响的大小是它们的差乘以光线能到达这里的概率 \(T_k\)。FIG 2 的解析梯度就是这个递推。

    Gradients. For each sample, \(\partial\hat C/\partial\mathbf c_k=w_k\) and \(\partial\hat C/\partial\alpha_k=T_k\,(\mathbf c_k-\mathbf C_{>k})\), where \(\mathbf C_{>k}\) is the colour blended from everything behind sample \(k\) (computed back to front, \(\mathbf C_{>k-1}=\alpha_k\mathbf c_k+(1-\alpha_k)\mathbf C_{>k}\), starting from the background). The intuition is plain: raising \(\alpha_k\) swaps the colour behind it for \(\mathbf c_k\), by the difference times the probability \(T_k\) that the ray gets there. FIG 2's analytic gradient is this recursion.

    NeRF：网络、损失、位置编码NeRF: network, loss, positional encoding

    $$ F_\Theta:\big(\gamma(\mathbf x),\gamma(\mathbf d)\big)\mapsto(\sigma,\mathbf c),\qquad \mathcal L(\Theta)=\sum_{\mathbf r\in\mathcal R}\big\|\hat C(\mathbf r;\Theta)-C_{\text{photo}}(\mathbf r)\big\|_2^2,\qquad \gamma(x)=\big(\sin(2^k\pi x),\cos(2^k\pi x)\big)_{k=0}^{L-1}. $$

    \(\sigma\) 只依赖位置 \(\mathbf x\)（保证多视角一致），颜色还依赖观察方向 \(\mathbf d\)（高光）。\(\mathcal R\) 是从已知位姿的照片里随机抽出的像素光线，位姿通常由 COLMAP（SfM + 束调整，11.6 节）给出，内参来自 10.3 节。原版 NeRF 对位置用 \(L=10\)、对方向用 \(L=4\)，所以 \(\gamma(\mathbf x)\in\R^{3\cdot2\cdot10}=\R^{60}\)。采样用分层采样 + 层次采样：先在光线上均匀（带抖动）取 64 点跑"粗"网络，再按粗权重 \(w_i\) 的分布重要性采样 128 点跑"细"网络，把采样集中在表面附近。训练是标准 SGD / Adam（6.4 节）。后续工作改进了不同方面：Mip-NeRF 360 / Zip-NeRF（抗锯齿与无界场景）、Instant-NGP（哈希网格 + 小 MLP，训练从小时降到分钟）。

    \(\sigma\) depends only on the position \(\mathbf x\) (guaranteeing multi-view consistency) while the colour also depends on the viewing direction \(\mathbf d\) (highlights). \(\mathcal R\) is a random batch of pixel rays from photographs with known poses, usually obtained from COLMAP (SfM + bundle adjustment, Lesson 11.6), with the intrinsics of Lesson 10.3. The original NeRF uses \(L=10\) for position and \(L=4\) for direction, so \(\gamma(\mathbf x)\in\R^{3\cdot2\cdot10}=\R^{60}\). Sampling is stratified, then hierarchical: 64 jittered points feed a "coarse" network, then 128 more are importance-sampled from the coarse weights \(w_i\) for the "fine" network, concentrating samples near surfaces. Training is ordinary SGD / Adam (Lesson 6.4). Later work improved different aspects: Mip-NeRF 360 / Zip-NeRF (anti-aliasing and unbounded scenes), Instant-NGP (hash grid + a tiny MLP, minutes instead of hours).

    3DGS：参数化、投影、混合、训练3DGS: parametrization, projection, blending, training

    每个高斯有 \(\boldsymbol\mu\in\R^3\)、尺度 \(\mathbf s\in\R^3_{>0}\)、旋转（单位四元数）、不透明度 \(o\in(0,1)\)、以及颜色（3 阶球谐，48 个系数，使颜色依赖视角）。协方差不直接优化，而是

    Each Gaussian has \(\boldsymbol\mu\in\R^3\), scales \(\mathbf s\in\R^3_{>0}\), a rotation (unit quaternion), an opacity \(o\in(0,1)\) and a colour (degree-3 spherical harmonics, 48 coefficients, which makes the colour view-dependent). The covariance is not optimized directly; instead

    $$ \hl{cov3}{\Sigma}=R\,S\,S^{\top}R^{\top}=(RS)(RS)^{\top}\ \Rightarrow\ \mathbf x^{\top}\Sigma\mathbf x=\|(RS)^{\top}\mathbf x\|^2\ge0,\qquad \Sigma\,R_{:,i}=s_i^2R_{:,i}. $$

    任何实数 \(R,S\) 都给出对称半正定矩阵，梯度更新永远不会产生"负方差"；直接对 \(\Sigma\) 的 9 个元素做梯度步则会很快跑出半正定锥。特征向量就是 \(R\) 的列、特征值是 \(s_i^2\)（1.11 节，FIG 4 读数里核对过）。实现里尺度存为 \(\log s\)，四元数每步归一化，不透明度存为 sigmoid 之前的值，保证约束自动满足。

    Any real \(R,S\) yields a symmetric positive semi-definite matrix, so a gradient step can never create a "negative variance", whereas stepping directly on the 9 entries of \(\Sigma\) soon leaves the PSD cone. The eigenvectors are the columns of \(R\) and the eigenvalues are \(s_i^2\) (Lesson 1.11; checked in FIG 4's readout). In code the scales are stored as \(\log s\), the quaternion is normalized each step and the opacity is stored before a sigmoid, so the constraints hold automatically.

    投影。设世界到相机的变换为 \(\mathbf x_c=W\mathbf x+\mathbf t\)（外参 \({}^{c}T_{w}\)，10.3 节），像素 \(\mathbf u=\pi(\mathbf x_c)=(f\,x/z+c_x,\ f\,y/z+c_y)\)。中心点精确投影 \(\boldsymbol\mu'=\pi(W\boldsymbol\mu+\mathbf t)\)；协方差按一阶泰勒展开 \(\pi(\mathbf x_c+\boldsymbol\epsilon)\approx\pi(\mathbf x_c)+J\boldsymbol\epsilon\) 传播（4.5 节）：

    Projection. With the world-to-camera transform \(\mathbf x_c=W\mathbf x+\mathbf t\) (the extrinsics \({}^{c}T_{w}\) of Lesson 10.3) and the pixel \(\mathbf u=\pi(\mathbf x_c)=(f\,x/z+c_x,\ f\,y/z+c_y)\), the centre projects exactly, \(\boldsymbol\mu'=\pi(W\boldsymbol\mu+\mathbf t)\); the covariance propagates through the first-order Taylor expansion \(\pi(\mathbf x_c+\boldsymbol\epsilon)\approx\pi(\mathbf x_c)+J\boldsymbol\epsilon\) (Lesson 4.5):

    $$ \hl{J}{J}=\frac{\partial\pi}{\partial\mathbf x_c}=\begin{bmatrix}f/z&0&-f\,x/z^2\\0&f/z&-f\,y/z^2\end{bmatrix}_{\mathbf x_c=W\boldsymbol\mu+\mathbf t},\qquad \hl{cov2}{\Sigma'}=J\,W\,\Sigma\,W^{\top}J^{\top}\in\R^{2\times2}. $$

    注意：这里的 \(\Sigma\) 在世界系，\(W\) 是旋转部分（平移不影响协方差）；\(J\) 在椭球中心处求值，所以远小于距离的椭球才准（FIG 4）。参考实现还在 \(\Sigma'\) 的对角线上加了一个很小的常数（0.3 像素²）当抗锯齿的低通滤波。屏幕上的高斯再按 FIG 2 的公式 \(\alpha_i=o_i\exp(-\tfrac12\,\Delta\mathbf u^{\top}\Sigma'^{-1}\Delta\mathbf u)\) 逐像素求值，并按相机系深度 \(z\) 从前到后混合。颜色 \(\mathbf c_i\) 由球谐函数在当前视线方向上求值。

    Note that \(\Sigma\) here is in the world frame and \(W\) is the rotation part (the translation does not affect a covariance), and \(J\) is evaluated at the ellipsoid's centre, so the formula is accurate only for ellipsoids much smaller than their distance (FIG 4). The reference implementation also adds a small constant (0.3 px²) to the diagonal of \(\Sigma'\) as an anti-aliasing low-pass filter. Each screen-space Gaussian is then evaluated per pixel with FIG 2's \(\alpha_i=o_i\exp(-\tfrac12\,\Delta\mathbf u^{\top}\Sigma'^{-1}\Delta\mathbf u)\) and blended front to back by camera-frame depth \(z\); the colour \(\mathbf c_i\) comes from the spherical harmonics evaluated along the current viewing direction.

    训练。损失 \(\mathcal L=(1-\lambda)\,\mathcal L_1+\lambda\,\mathcal L_{\text{D-SSIM}}\)（\(\lambda=0.2\)），初始高斯来自 SfM 的稀疏点云（12.1 节的点，被赋予了"大小"）。每隔若干步做自适应密度控制：累计视图空间位置梯度的平均大小超过阈值的高斯被稠密化——大的分裂（尺度缩小到约 1/1.6，位置从原高斯采样），小的克隆；不透明度过低或尺寸过大的被剪掉；并周期性地重置不透明度以清除漂浮物。渲染用按 16×16 像素分块的 GPU 光栅化器，对"高斯 × 分块"的键值（分块号 + 深度）排序后逐块混合，梯度也通过同一个分块结构反传。FIG 2 是它的二维、L2 损失版本，其余逻辑一致。

    Training. The loss is \(\mathcal L=(1-\lambda)\,\mathcal L_1+\lambda\,\mathcal L_{\text{D-SSIM}}\) (\(\lambda=0.2\)), and the initial Gaussians come from the sparse SfM point cloud (the points of Lesson 12.1, given a size). Every so often adaptive density control runs: Gaussians whose average view-space positional gradient exceeds a threshold are densified, big ones split (scales shrunk by about 1.6, positions sampled from the parent) and small ones cloned; those with very low opacity or excessive size are pruned, and opacities are reset periodically to clear floaters. Rendering uses a GPU rasterizer over 16×16-pixel tiles: key-value pairs (tile id + depth) for each Gaussian-tile overlap are sorted, then each tile blends its list, and the backward pass reuses the same tile structure. FIG 2 is its 2-D, L2-loss version with the same logic.

    与本章其他表示的关系Relation to the other representations of this chapter

    点云（12.1）：3DGS 是"每个点带了形状、不透明度和朝向相关颜色"的点云，初始化就来自 SfM 点。SDF 与网格（12.2）：NeRF 的密度可以改成由 SDF 导出（NeuS、VolSDF），这样体渲染学出的是一个有明确表面的隐式几何。渲染管线（12.3）：3DGS 的渲染就是一个自定义的光栅化管线（投影 → 分块 → 深度排序 → alpha 混合），NeRF 则是光线投射；二者共用同一个 alpha 混合公式。

    Point clouds (12.1): 3DGS is a point cloud whose points carry a shape, an opacity and view-dependent colour, initialized from the SfM points. SDFs and meshes (12.2): NeRF's density can be derived from an SDF (NeuS, VolSDF), so that volume rendering learns an implicit geometry with a well-defined surface. The graphics pipeline (12.3): 3DGS rendering is a custom rasterization pipeline (project, tile, depth sort, alpha blend) while NeRF is ray casting; both share the same alpha-blending formula.

    6机器人中的它：渲染要多快，才能进控制环？In robotics: how fast must rendering be to enter a control loop?

    在机器人里，辐射场往往不是"离线做一份漂亮的模型"就结束了：SLAM 要在每一帧用渲染误差修正位姿，仿真和策略学习要渲染成千上万张图，规划器要反复查询它。所以渲染的算术量和延迟决定了它能不能进入闭环。下面用一个粗略的成本模型把 NeRF 和 3DGS 摆在一起：它只算浮点运算的数量级，真实速度还受显存带宽、排序和实现影响（3DGS 的实际时间主要花在排序与内存访问上，所以按 FLOP 算出的毫秒数只是下界）。

    In robotics a radiance field is rarely a one-off "pretty model built offline": SLAM corrects the pose from the rendering error on every frame, simulation and policy learning render thousands of images, and planners query it repeatedly. So the arithmetic cost and latency of rendering decide whether it can enter a closed loop. The crude cost model below sets NeRF and 3DGS side by side. It counts only the orders of magnitude of floating-point operations; the real speed also depends on memory bandwidth, sorting and the implementation (the real time of 3DGS goes mostly to sorting and memory access, so the milliseconds computed from FLOPs are a lower bound).

    FIG 5渲染一帧的算术量：光线步进 vs 光栅化Arithmetic per frame: ray marching vs rasterization
    条形是每帧浮点运算量（对数坐标，\(10^8\) 到 \(10^{15}\)）。假设（都是粗略的）：原版 NeRF 每个采样的 MLP 约 0.6 M 次乘加；哈希网格版本的小 MLP 约 \(2\times10^4\) FLOP，且空区域跳过后只剩约 10% 的采样；3DGS 每个高斯预处理约 300 FLOP、排序键约 6 个分块×30 FLOP，每个像素混合约 50 个高斯×25 FLOP。时间是在持续 10 TFLOP/s 下的纯算术下界。 Bars show floating-point operations per frame (log scale, \(10^8\) to \(10^{15}\)). Assumptions (all crude): the original NeRF's MLP costs about 0.6 M multiply-adds per sample; a hash-grid variant's tiny MLP about \(2\times10^4\) FLOP, with empty-space skipping leaving about 10% of the samples; 3DGS about 300 FLOP per Gaussian for preprocessing, about 6 tiles × 30 FLOP of sort keys per Gaussian, and about 50 Gaussians × 25 FLOP of blending per pixel. The times are the pure-arithmetic lower bound at a sustained 10 TFLOP/s.
    试一试Try this
    • 选 1080p、每条光线 192 个采样、100 万高斯，对比原版 NeRF 与 3DGS 的条形：差了几个数量级？再把高斯数拉到 \(10^7\)，差距缩小多少？Choose 1080p, 192 samples per ray and 106 Gaussians and compare the original NeRF's bar with 3DGS's: how many orders of magnitude apart? Then push the Gaussians to \(10^7\): how much does the gap shrink?
    真实系统里In real systems

    工具：nerfstudio（模块化的 NeRF / 高斯训练框架）、gsplat（高斯泼溅的 CUDA 库）、INRIA 的原版 3DGS 实现、Instant-NGP；相机位姿通常来自 COLMAP。建图与 SLAM：iMAP（Sucar 等，ICCV 2021）和 NICE-SLAM（Zhu 等，CVPR 2022）把神经场当地图；SplaTAM（Keetha 等，CVPR 2024）和 MonoGS / Gaussian Splatting SLAM（Matsuki 等，CVPR 2024）用高斯当地图并以渲染误差跟踪相机，GS-ICP SLAM 把 G-ICP 与高斯地图结合。操作：Dex-NeRF（Ichnowski 等，CoRL 2021）用 NeRF 恢复透明物体几何来抓取；Evo-NeRF、GraspNeRF 等在辐射场里估计抓取；LERF（Kerr 等，ICCV 2023）把语言特征蒸馏进辐射场，F3RM（Shen 等，CoRL 2023）把类似的特征场用于语言指令的抓取。仿真与数据：用手机拍摄重建场景、再导入仿真（real2sim），或用新视角合成做策略学习的数据增强，目前是活跃方向，各仿真器的导入支持不同，请查最新文档。以上多数是研究原型：具体精度、速度和适用条件请以原论文为准。

    Tools: nerfstudio (a modular NeRF / Gaussian training framework), gsplat (a CUDA library for Gaussian splatting), the original INRIA 3DGS implementation, Instant-NGP; camera poses usually come from COLMAP. Mapping and SLAM: iMAP (Sucar et al., ICCV 2021) and NICE-SLAM (Zhu et al., CVPR 2022) use neural fields as the map; SplaTAM (Keetha et al., CVPR 2024) and MonoGS / Gaussian Splatting SLAM (Matsuki et al., CVPR 2024) use Gaussians as the map and track the camera by rendering error, and GS-ICP SLAM combines G-ICP with a Gaussian map. Manipulation: Dex-NeRF (Ichnowski et al., CoRL 2021) recovers the geometry of transparent objects with a NeRF for grasping; Evo-NeRF, GraspNeRF and others estimate grasps inside a radiance field; LERF (Kerr et al., ICCV 2023) distills language features into a radiance field and F3RM (Shen et al., CoRL 2023) uses a similar feature field for language-conditioned grasping. Simulation and data: reconstructing a scene from phone footage and importing it into a simulator (real2sim), or using novel-view synthesis to augment data for policy learning, are active directions; import support differs between simulators, so check the current documentation. Most of the above are research prototypes: consult the original papers for accuracy, speed and conditions of use.

    接下来读什么。这一节也是第三部分的终点：第 10 章让你知道一个像素从哪里来，第 11 章让你从多张图恢复相机和结构，第 12 章让你决定怎样表示结果——点云、体素 / 网格 / SDF、渲染管线，以及可微渲染的辐射场。后面：第四部分（状态估计与 SLAM）把这里"事先由 SfM 给定"的相机位姿改成在线估计，并把 EKF 的雅可比技巧用到整个系统；第五部分（神经网络基础）把 FIG 3 里当黑盒的 MLP、反向传播、自动微分和位置编码讲清楚，之后你才能自己训练 NeRF 或高斯模型；第七部分（基础模型与世界模型）再回到"机器人脑中的世界长什么样"这个问题。

    What comes next. This lesson closes Part III: Chapter 10 told you where a pixel comes from, Chapter 11 how to recover cameras and structure from several images, and Chapter 12 how to represent the result: point clouds, voxels / meshes / SDFs, the graphics pipeline, and differentiable radiance fields. Ahead: Part IV (state estimation and SLAM) turns the camera poses we assumed "given by SfM beforehand" into quantities estimated online, and uses the EKF's Jacobian trick on the whole system; Part V (neural-network foundations) explains the MLP, backpropagation, automatic differentiation and positional encoding that FIG 3 treated as a black box, which you will need to train a NeRF or Gaussian model yourself; Part VII (foundation models and world models) returns to the question of what the world looks like inside a robot's head.

    7常见误区Common pitfalls

    误区 1：照片够多就行，位姿差一点没关系Pitfall 1: enough photos is enough, slightly wrong poses do not matter

    两种方法都把相机位姿当作已知真值。位姿有误差（SfM 在弱纹理、重复纹理、快速运动下失败；轮式里程计漂移）时，同一个表面点在不同照片里"对不上"，优化器只能把它抹成模糊的一团或分裂成漂浮物（floaters）。实践里先用 11.6 节的重投影误差检查 SfM 的质量，再训练；SLAM 类方法则把位姿也作为变量联合优化。

    Both methods treat the camera poses as known ground truth. With pose error (SfM failing on weak or repetitive texture and fast motion; wheel odometry drifting), the same surface point does not line up across photographs and the optimizer can only smear it into a blur or split it into floaters. In practice check SfM quality through the reprojection error of Lesson 11.6 before training; SLAM-style methods optimize the poses jointly as variables.

    误区 2：场景是静止的Pitfall 2: the scene is static

    辐射场假设所有照片看到的是同一个静止的世界。拍摄时有人走过、机械臂在动、光照在变，同一个位置在不同照片里颜色矛盾，模型要么把它平均成雾，要么留下"鬼影"。机器人自己的手臂和影子也算动态物体。常见对策是给动态物体打掩码、或用显式建模动态的变体，但默认的 NeRF / 3DGS 做不到。

    A radiance field assumes all photographs show the same static world. If a person walks through, the arm moves or the lighting changes during capture, the same location has contradictory colours in different photographs and the model either averages them into haze or leaves "ghosts". The robot's own arm and shadow count as dynamic objects. Usual remedies mask the dynamic objects or use variants that model them explicitly; plain NeRF / 3DGS cannot.

    误区 3：在训练视角上看着很好，就是好Pitfall 3: it looks great on the training views, so it is good

    FIG 3 已经演示了原理：训练损失可以趋近 0，而样本之间一团糟。辐射场同理：它被训练去复现训练照片，在训练视角附近可以很逼真，离开这些视角就会出现漂浮物、"空洞"和过平滑。评估必须用留出的视角（测试集）；对机器人更要问："我真正会去的视角，覆盖到了吗？"

    FIG 3 already showed the principle: the training loss can go to 0 while everything between samples is a mess. A radiance field is the same: it is trained to reproduce the training photographs, can look perfect near those viewpoints and shows floaters, "holes" and over-smoothing away from them. Evaluation needs held-out views; for a robot, also ask "have I covered the viewpoints I will actually visit?"

    误区 4：照片级真实 = 几何准确，可以直接做碰撞检测Pitfall 4: photorealistic means accurate geometry, so I can collision-check against it

    渲染得像不等于几何对：NeRF 的密度可以在玻璃、反光面处"作弊"，3DGS 的高斯是一团半透明椭球，不是水密表面，也没有"内外"。要做碰撞检测或抓取，需要先转成网格 / TSDF / ESDF（12.2 节）或用带几何正则的变体（2DGS、NeuS 类）。另外单目 SfM 的尺度是任意的，度量尺度要靠已知基线、IMU 或深度传感器恢复。

    Looking right does not mean the geometry is right: a NeRF's density can "cheat" at glass and shiny surfaces, and 3DGS's Gaussians are a cloud of translucent ellipsoids, not a watertight surface, with no inside or outside. Collision checking or grasping needs a conversion to a mesh / TSDF / ESDF (Lesson 12.2) or a variant with geometric regularization (2DGS, NeuS-style). Also, the scale of monocular SfM is arbitrary; metric scale has to come from a known baseline, an IMU or a depth sensor.

    误区 5：一阶近似和排序"差不多就行"Pitfall 5: the first-order approximation and the sort are "close enough"

    FIG 4 说明 \(\Sigma'=JW\Sigma W^{\top}J^{\top}\) 只在椭球远小于其深度时准确；靠近相机的大高斯会被投成错误的形状，这是 3DGS 近景伪影的来源之一。另外 3DGS 按每个高斯一个深度排序，而不是每个像素；两个重叠的大高斯在视角变化时顺序翻转，画面会"跳变"（popping）。还有内存：每个高斯约 59 个浮点数，几百万个高斯就是几百 MB 到 GB，机器人板载 GPU 不一定放得下。

    FIG 4 shows that \(\Sigma'=JW\Sigma W^{\top}J^{\top}\) is accurate only when the ellipsoid is much smaller than its depth; a big Gaussian close to the camera is projected to the wrong shape, one source of 3DGS close-range artifacts. Also, 3DGS sorts by one depth per Gaussian, not per pixel; when two overlapping large Gaussians swap order as the view changes the image "pops". And there is memory: about 59 floats per Gaussian means hundreds of MB to GBs for millions of Gaussians, which an onboard robot GPU may not hold.

    8小练习Check yourself

    1. 光线上某一段：\(\sigma=2\ \text{m}^{-1}\)，宽 \(\delta=0.5\ \text{m}\)，光线到达这一段起点的透射率 \(T_i=0.5\)。这一段的权重 \(w_i=T_i\alpha_i\) 是多少？1. On one interval of a ray, \(\sigma=2\ \text{m}^{-1}\), \(\delta=0.5\ \text{m}\), and the transmittance reaching the interval's start is \(T_i=0.5\). What is its weight \(w_i=T_i\alpha_i\)?
    \(\alpha=1-e^{-\sigma\delta}=1-e^{-1}\approx0.632\)，\(w=T\alpha=0.5\times0.632\approx0.316\)。0.632 只是这一段"自己拦下光线"的概率，还要乘上光线能走到这里的概率 \(T_i\)。朴素的 \(\sigma\delta=1.0\) 则会让 \(\alpha\) 达到 1，是 FIG 1 里"朴素"模式过曝的原因。\(\alpha=1-e^{-\sigma\delta}=1-e^{-1}\approx0.632\), \(w=T\alpha=0.5\times0.632\approx0.316\). The 0.632 is only the probability that this interval stops the ray by itself; it must be multiplied by the probability \(T_i\) that the ray gets here. The naive \(\sigma\delta=1.0\) would make \(\alpha=1\), the reason "naive" over-exposes in FIG 1.
    2. 3DGS 为什么把协方差写成 \(\Sigma=RSS^{\top}R^{\top}\)，而不是直接优化 \(\Sigma\) 的元素？2. Why does 3DGS write the covariance as \(\Sigma=RSS^{\top}R^{\top}\) instead of optimizing the entries of \(\Sigma\) directly?
    \(\mathbf x^{\top}\Sigma\mathbf x=\|(RS)^{\top}\mathbf x\|^2\ge0\) 对任何实数 \(R,S\) 成立；对 9 个元素直接做梯度步，很容易得到有负特征值的矩阵，那就不是协方差了。\(\Sigma\) 一般不是对角阵（\(R\) 是任意旋转），渲染当然依赖相机（通过 \(W\) 和 \(J\)）。\(\mathbf x^{\top}\Sigma\mathbf x=\|(RS)^{\top}\mathbf x\|^2\ge0\) for any real \(R,S\); a direct gradient step on 9 entries easily yields a matrix with a negative eigenvalue, which is no longer a covariance. \(\Sigma\) is generally not diagonal (\(R\) is an arbitrary rotation), and rendering certainly depends on the camera (through \(W\) and \(J\)).
    3. 原版 NeRF 对三维位置 \((x,y,z)\) 用 \(L=10\) 的傅里叶编码（不含原始坐标本身）。\(\gamma(\mathbf x)\) 的维数是？3. The original NeRF applies a Fourier encoding with \(L=10\) to the 3-D position \((x,y,z)\) (not counting the raw coordinates). What is the dimension of \(\gamma(\mathbf x)\)?
    每个坐标有 \(L\) 个频率，每个频率一对 sin / cos：\(3\times10\times2=60\)。最高频率 \(2^{9}\pi\) 对应的周期是 \(2/512\approx0.004\)（坐标范围是 \([-1,1]\)），也就是 FIG 3 里 \(L\) 增大时能画出越来越细的结构。Each coordinate has \(L\) frequencies, each with a sin / cos pair: \(3\times10\times2=60\). The highest frequency \(2^{9}\pi\) has a period of \(2/512\approx0.004\) (the coordinate range is \([-1,1]\)), which is why FIG 3 draws finer and finer structure as \(L\) grows.
    4. 一个球对称高斯 \(\Sigma=0.01\,I\)（世界系，\(W=I\)），中心在相机系 \((x,y,z)=(1,0,4)\)，焦距 \(f=400\) 像素。\(\Sigma'\) 的 \(u\) 方向方差 \(\Sigma'_{uu}\) 是多少？4. A spherical Gaussian \(\Sigma=0.01\,I\) (world frame, \(W=I\)) is centred at camera-frame \((x,y,z)=(1,0,4)\) with focal length \(f=400\) px. What is the \(u\)-direction variance \(\Sigma'_{uu}\)?
    \(J\) 的第一行 \((f/z,\,0,\,-fx/z^2)=(100,\,0,\,-25)\)，所以 \(\Sigma'_{uu}=0.01\,(100^2+25^2)=106.25\ \text{px}^2\)。若只看 \(f/z\) 会得到 100，漏掉的 25 来自"物体偏离光轴时，深度方向的不确定性也会投到 \(u\) 上"，这正是 \(J\) 第三列的作用。The first row of \(J\) is \((f/z,\,0,\,-fx/z^2)=(100,\,0,\,-25)\), so \(\Sigma'_{uu}=0.01\,(100^2+25^2)=106.25\ \text{px}^2\). Looking only at \(f/z\) gives 100; the missing 25 comes from "when the object is off the optical axis, uncertainty along the depth direction also projects onto \(u\)", the job of \(J\)'s third column.
    5. 你用手机绕桌子拍了一圈来重建场景，但拍摄过程中有一个人在桌子后面来回走动。训练出的辐射场最可能怎样？5. You walk a phone around a table to reconstruct the scene, but someone keeps walking back and forth behind the table during capture. What will the trained radiance field most likely show?
    静态场景假设被打破：同一个位置在不同照片里有时是人、有时是背景，一个静止的场只能折中，表现为半透明的鬼影或浮在空中的色块。需要显式的动态建模、对动态物体加掩码，或者让拍摄时场景静止。The static-scene assumption is broken: the same location is sometimes a person and sometimes background in different photographs, and a static field can only compromise, appearing as translucent ghosts or colour blobs floating in the air. Use explicit dynamic modelling, mask the dynamic objects, or keep the scene still during capture.

    9延伸阅读Further reading

    • B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi & R. Ng, NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, ECCV 2020 — NeRF 原文：体渲染、位置编码、层次采样The NeRF paper: volume rendering, positional encoding, hierarchical sampling
    • B. Kerbl, G. Kopanas, T. Leimkühler & G. Drettakis, 3D Gaussian Splatting for Real-Time Radiance Field Rendering, ACM Trans. Graphics (SIGGRAPH) 42(4), 2023 — 3DGS 原文：参数化、分块光栅化、自适应密度控制The 3DGS paper: parametrization, tile rasterizer, adaptive density control
    • M. Zwicker, H. Pfister, J. van Baar & M. Gross, EWA Volume Splatting, IEEE Visualization 2001 — 协方差经雅可比投影的出处The source of projecting a covariance through the Jacobian
    • N. Max, Optical Models for Direct Volume Rendering, IEEE Trans. Visualization and Computer Graphics 1(2), 1995 — 体渲染方程与 alpha 合成求积The volume rendering equation and its alpha-compositing quadrature
    • M. Tancik et al., Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS 2020 — 位置编码为什么有效（FIG 3 的理论背景）Why positional encoding works (the theory behind FIG 3)
    • N. Rahaman et al., On the Spectral Bias of Neural Networks, ICML 2019 — 谱偏置Spectral bias
    • J. T. Barron et al., Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields, CVPR 2022; Zip-NeRF, ICCV 2023 — 抗锯齿与无界场景的高质量 NeRFHigh-quality anti-aliased NeRFs for unbounded scenes
    • T. Müller, A. Evans, C. Schied & A. Keller, Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, ACM Trans. Graphics (SIGGRAPH) 2022 — 哈希网格：把 NeRF 训练缩短到分钟Hash grids: NeRF training in minutes
    • A. Tewari et al., Advances in Neural Rendering, Computer Graphics Forum (EUROGRAPHICS STAR) 2022 — 神经渲染综述A survey of neural rendering
    • P. Wang et al., NeuS, NeurIPS 2021; L. Yariv et al., VolSDF, NeurIPS 2021 — 用 SDF 做体渲染，得到明确的表面（与 12.2 节相连）Volume rendering with an SDF to get a well-defined surface (links to Lesson 12.2)
    • J. L. Schönberger & J.-M. Frahm, Structure-from-Motion Revisited, CVPR 2016 — COLMAP：为辐射场提供相机位姿COLMAP: the camera poses that radiance fields start from
    • M. Tancik et al., Nerfstudio: A Modular Framework for Neural Radiance Field Development, SIGGRAPH 2023 — 上手 NeRF / 高斯的开源框架An open-source framework for getting started with NeRFs and Gaussians