3D 图形学渲染管线基础3D Graphics Pipeline Basics

机器人学里到处都有渲染器。仿真器（Isaac Sim、MuJoCo、Gazebo、PyBullet）要给仿真里的机器人"拍照"，让视觉策略能在虚拟世界里训练和测试。合成数据让感知模型不必人工标注：渲染器可以免费地给出 RGB、深度图和每个像素属于哪个物体（实例分割）。可视化工具（RViz 等）要把机器人每一节连杆的网格放到当前关节角对应的位置。可微渲染与神经渲染（12.4 的 NeRF 与 3D Gaussian Splatting）把"渲染"放进优化循环，通过比较渲染图与真实图像来恢复位姿和场景。

渲染是本部分其余内容的逆过程。第 10–11 章从图像出发，估计相机、深度和三维点；渲染则从三维模型和相机出发，生成图像。相机数学完全一样：10.1 的针孔模型、10.3 的内参 \(K\) 和外参 \({}^{c}T_{w}\)。多出来的只有两件事：可见性（同一个像素上有好几个表面，哪个在前面？）和着色（这个表面在这个光照下是什么颜色？）。

本课走一遍实时图形学里最经典的做法，光栅化管线：先用矩阵链把每个顶点从模型坐标变到屏幕像素（FIG 1、FIG 4），再逐像素地判断三角形盖住了谁、深度是多少、属性怎么插值（FIG 2、FIG 3），最后着色并与光线追踪对照（FIG 5），并用它生成机器人训练数据（FIG 6）。12.4 里的 NeRF 沿相机光线积分、3D Gaussian Splatting 则是把高斯椭球"光栅化"到屏幕上，两者都建立在这里的思想之上。

Renderers are everywhere in robotics. Simulators (Isaac Sim, MuJoCo, Gazebo, PyBullet) have to "take photos" for the simulated robot so that vision policies can be trained and tested in a virtual world. Synthetic data frees perception models from hand labelling: a renderer gives RGB, depth and the object each pixel belongs to (instance segmentation) for free. Visualizers (RViz and friends) place every link mesh of the robot at the position implied by the current joint angles. Differentiable and neural rendering (NeRF and 3D Gaussian Splatting in 12.4) put rendering inside an optimization loop, recovering poses and scenes by comparing rendered with real images.

Rendering is the inverse of the rest of this Part. Chapters 10–11 start from images and estimate cameras, depth and 3D points; rendering starts from a 3D model and a camera and produces an image. The camera math is identical: the pinhole model of 10.1, the intrinsics \(K\) and extrinsics \({}^{c}T_{w}\) of 10.3. Only two things are added: visibility (several surfaces project to the same pixel; which is in front?) and shading (what colour does this surface have under this light?).

This lesson walks through the classic approach of real-time graphics, the rasterization pipeline: a chain of matrices takes every vertex from model coordinates to screen pixels (FIG 1, FIG 4), then a per-pixel stage decides which triangle covers which pixel, at what depth, and how attributes are interpolated (FIG 2, FIG 3); finally we shade and compare with ray tracing (FIG 5) and use the whole thing to generate robot training data (FIG 6). NeRF in 12.4 integrates along camera rays and 3D Gaussian Splatting "rasterizes" Gaussian blobs onto the screen; both build on the ideas here.

一个三维模型就是一堆三角形，每个三角形有三个顶点。渲染的第一半工作是把每个顶点变成屏幕上的一个像素位置，办法是依次换坐标系，每一步都是你熟悉的一个矩阵：

1. 模型 → 世界：模型矩阵 \(M\)。每个网格在自己的坐标系里建模（连杆的原点就在它的关节上），\(M\) 把它摆到世界里。对机器人，\(M_i = {}^{w}T_i(\boldsymbol\theta)\) 就是 7.3 的正运动学：第 \(i\) 个连杆网格的位姿。
2. 世界 → 眼（相机）：视图矩阵 \(V\)，即相机位姿的逆（10.3 的外参），再加一次坐标轴翻转。
3. 眼 → 裁剪：投影矩阵 \(P\)。它是一个 \(4\times4\) 矩阵，把"视锥"（相机看得见的那块棱台）变形，同时让第四个分量 \(w_c\) 等于深度。
4. 裁剪 → NDC：透视除法，每个分量除以 \(w_c\)。这一步才产生透视（远处变小），并把视锥压成立方体 \([-1,1]^3\)。
5. NDC → 屏幕：视口变换，把立方体拉伸成 \(w\times h\) 的像素窗口，深度放进 \([0,1]\)。

在 FIG 1 里亲手走一遍。机械臂是 7.3 的两连杆加底座和一个方块，每个部件都有自己的模型矩阵。拖动"阶段"滑块，顶点会在 6 个空间之间连续变形；紫色框是相机的视锥（在每个空间里它长什么样）。右边是同一条管线最终渲染出的图像。琥珀色的点 \(p\) 是连杆 2 末端的一个顶点，读数列出它在每个空间里的坐标。

A 3D model is a pile of triangles with three vertices each. The first half of rendering turns every vertex into a pixel position by changing coordinates step by step; every step is a matrix you already know:

1. model → world: the model matrix \(M\). Each mesh is authored in its own frame (a link's origin sits at its joint) and \(M\) places it in the world. For a robot, \(M_i = {}^{w}T_i(\boldsymbol\theta)\) is exactly the forward kinematics of 7.3: the pose of link \(i\)'s mesh.
2. world → eye (camera): the view matrix \(V\), the inverse of the camera pose (the extrinsics of 10.3) plus an axis flip.
3. eye → clip: the projection matrix \(P\). A \(4\times4\) matrix that deforms the "view frustum" (the truncated pyramid the camera sees) and sets the fourth component \(w_c\) equal to the depth.
4. clip → NDC: the perspective divide, every component divided by \(w_c\). This is the step that creates perspective (far things shrink) and squeezes the frustum into the cube \([-1,1]^3\).
5. NDC → screen: the viewport transform stretches the cube to the \(w\times h\) pixel window and puts depth in \([0,1]\).

Walk through it in FIG 1. The arm is the two-link arm of 7.3 with a base and a cube, and each part has its own model matrix. Drag the "stage" slider and the vertices morph continuously between the six spaces; the purple box is the camera's view volume (what it looks like in every space). On the right is the image the same pipeline finally renders. The amber dot \(p\) is one vertex at the tip of link 2, and the readout lists its coordinates in every space.

顶点到了屏幕坐标，下一步要回答的是：哪些像素被这个三角形盖住了？光栅化器的回答很朴素：对三角形包围盒里的每个像素，检查像素中心在不在三角形里。检查用的是边函数。对有向边 \(a\to b\)，

就是 1.8 的二维叉积，等于三角形 \((a,b,p)\) 有向面积的两倍。点 \(\mathbf p\) 在三角形内，当且仅当三条边的 \(E\) 符号一致。而三个子三角形的面积之比正是重心坐标 \(\hl{lam}{\lambda_0,\lambda_1,\lambda_2}\)（和为 1，在内部都大于 0）。有了它，三角形上任何量都可以从顶点值插值出来：\(f(\mathbf p)=\sum_i\lambda_i f_i\)，不管 \(f\) 是颜色、深度还是纹理坐标。

FIG 2 是一台真正在页面里跑的小型软件光栅化器（64×48 像素，放大后每个像素都看得见）。拖动三个顶点；移动鼠标选取"探针"像素，右边的读数给出它的边函数、\(\lambda\) 和插值结果。右图是深度缓冲（越亮越近）。把顶点深度 \(z_0,z_1,z_2\) 设得不同，深度图就是同一组 \(\lambda\) 插值出来的渐变。

Once the vertices are in screen coordinates the next question is: which pixels does this triangle cover? The rasterizer's answer is plain: for every pixel in the triangle's bounding box, test whether the pixel centre lies inside. The test uses edge functions. For a directed edge \(a\to b\),

is the 2D cross product of 1.8, twice the signed area of the triangle \((a,b,p)\). A point \(\mathbf p\) is inside iff the three \(E\)'s agree in sign, and the ratios of the three sub-triangle areas are the barycentric coordinates \(\hl{lam}{\lambda_0,\lambda_1,\lambda_2}\) (summing to 1, all positive inside). With them anything defined at the vertices can be interpolated over the triangle, \(f(\mathbf p)=\sum_i\lambda_i f_i\), be it colour, depth or texture coordinates.

FIG 2 is a real software rasterizer running in the page (64×48 pixels, magnified so that every pixel is visible). Drag the three vertices; move the mouse to choose the "probe" pixel and the readout shows its edge functions, its \(\lambda\) and the interpolated value. The right image is the depth buffer (brighter = nearer). Give the vertices different depths \(z_0,z_1,z_2\) and the depth image is the same \(\lambda\) interpolating a gradient.

z-buffer 每个像素只留一个"最近"的片元，所以只适合不透明表面。半透明片元带一个不透明度 \(\alpha\in[0,1]\)，它挡住后面 \(\alpha\) 比例的光，自己贡献 \(\alpha\,\mathbf c\)。从后往前画时，每个新片元用"over"运算盖在已有颜色上：\(C\leftarrow\alpha\,\mathbf c+(1-\alpha)\,C\)，这就是画家算法加上混合。也可以从前往后累积，同时记下还剩多少光能穿过去（透射率 \(T\)，初值 1）：\(C\leftarrow C+T\alpha\,\mathbf c\)，\(T\leftarrow T(1-\alpha)\)。两种顺序结果相同，展开后都是 \(C=\sum_i \mathbf c_i\alpha_i\prod_{j<i}(1-\alpha_j)\)（下标从近到远）。从前往后的好处是 \(T\) 小到可以忽略时就能提前停下。无论哪种顺序，片元都必须按深度排好，这正是半透明物体不能交给 z-buffer 的原因。12.4 的 NeRF 和 3D Gaussian Splatting 用的就是这个从前往后的公式。

A z-buffer keeps a single "nearest" fragment per pixel, so it only suits opaque surfaces. A translucent fragment carries an opacity \(\alpha\in[0,1]\): it blocks a fraction \(\alpha\) of the light behind it and contributes \(\alpha\,\mathbf c\) itself. Drawing back to front, each new fragment is laid over the colour so far with the "over" operator, \(C\leftarrow\alpha\,\mathbf c+(1-\alpha)\,C\): the painter's algorithm plus blending. One can also accumulate front to back while tracking how much light still gets through (the transmittance \(T\), starting at 1): \(C\leftarrow C+T\alpha\,\mathbf c\), \(T\leftarrow T(1-\alpha)\). Both orders give the same result, which expands to \(C=\sum_i \mathbf c_i\alpha_i\prod_{j<i}(1-\alpha_j)\) (index running near to far); front to back has the advantage that it can stop early once \(T\) is negligible. Either way the fragments must be sorted by depth, which is why translucent objects cannot be left to the z-buffer. NeRF and 3D Gaussian Splatting in 12.4 use exactly this front-to-back formula.

重心坐标是在屏幕上算的，可是颜色、纹理坐标这些属性是定义在三维表面上的。屏幕上的等距不等于表面上的等距（远处的一段距离在屏幕上更短），所以直接用屏幕空间的 \(\lambda_i\) 线性插值纹理坐标是错的。FIG 3 渲染一块铺满棋盘格的地面（OpenCV 相机，地面在相机下方，每格 0.45 m）：左图用屏幕空间的 \(\lambda\) 直接插值纹理坐标，中图用透视校正插值，右图是真值，即对每个像素发出一条光线与地面求交。线性插值的格子沿深度方向被拉得一样大，并在两个三角形的对角线处出现折痕。

Barycentric coordinates are computed on the screen, but attributes such as colour and texture coordinates live on the 3D surface. Equal steps on the screen are not equal steps on the surface (a distance far away covers fewer pixels), so interpolating texture coordinates directly with the screen-space \(\lambda_i\) is wrong. FIG 3 renders a checkerboard floor (an OpenCV camera above the floor, 0.45 m squares): the left image interpolates texture coordinates directly with the screen-space \(\lambda\), the middle one uses perspective-correct interpolation, the right one is the truth, obtained by casting a ray through every pixel to the floor. The linear version draws all squares equally large along the depth direction and shows a crease along the diagonal between the two triangles.

在 10.1 和 10.3 里，相机用内参 \(K\)（\(f_x,f_y,c_x,c_y\)）和图像尺寸描述，像素 \((u,v)\) 的原点在左上角，\(v\) 向下，相机看向 \(+z\)。图形学则用视场角、宽高比、近远平面构造 \(P\)，眼空间里相机看向 \(-z\)，\(y\) 向上，NDC 的 \(y\) 向上。两套语言描述同一台相机，可以互相换算。FIG 4 同时显示两边：拖动视场角和主点偏移，看 \(K\) 与 \(P\) 的各项如何一起变；拖动琥珀色的点，两条路线（OpenCV 的 \(u=f_xx/z+c_x\)，对 OpenGL 的 \(P\,V\,\tilde{\mathbf p}\) 再 ÷\(w\) 再视口变换）给出同一个像素。右图是深度缓冲值 \(d(z)\)：它是 \(z\) 的非线性函数，近平面附近占掉了绝大部分数值范围。

In 10.1 and 10.3 a camera is described by the intrinsics \(K\) (\(f_x,f_y,c_x,c_y\)) and the image size; the pixel \((u,v)\) has its origin at the top-left with \(v\) down, and the camera looks along \(+z\). Graphics builds \(P\) from a field of view, an aspect ratio and the near and far planes; in eye space the camera looks along \(-z\) with \(y\) up, and NDC's \(y\) is up too. Two languages describe the same camera and they convert into one another. FIG 4 shows both sides at once: drag the field of view and the principal-point offset and watch the entries of \(K\) and \(P\) change together; drag the amber point and the two routes (OpenCV's \(u=f_xx/z+c_x\), versus OpenGL's \(P\,V\,\tilde{\mathbf p}\), then \(\div w\), then the viewport transform) give the same pixel. The right plot is the depth-buffer value \(d(z)\): a non-linear function of \(z\) that spends most of its numeric range close to the near plane.

可见性解决了，剩下的是颜色。最简单的局部光照模型把表面颜色拆成三项：环境光（一个常数，代替无处不在的间接光）、漫反射（Lambert：与 \(\mathbf n\cdot\mathbf l\) 成正比，正是 1.7 的点积，法向和光线夹角越小越亮）和镜面反射（Blinn–Phong：\((\mathbf n\cdot\mathbf h)^m\)，\(\mathbf h\) 是光线方向与视线方向的半程向量，\(m\) 越大高光越小）。法向 \(\mathbf n\) 来自网格：每个面一个法向给出平面着色（看得见棱角）；每个顶点一个法向再插值，则网格看上去是光滑的。把光照在顶点算好再插值颜色叫 Gouraud 着色，把法向插值后逐像素算光照叫 Phong 着色，后者才能让高光落在两个顶点之间。

FIG 5 用一个球面网格和一块棋盘格地面比较两种渲染。左图是上面的光栅化管线；中图是光线追踪：每个像素发一条光线，与解析球面和平面求交，再用一条朝光源的阴影光线判断该点是否被球挡住；右图是两者之差（放大 4 倍）。把球面细分 \(N\) 加大，网格逼近真正的球，两图越来越接近，差别只剩多边形轮廓。打开阴影，地面上的影子在差异图里亮起，因为光栅化器只知道"这个片元的法向和光向"，不知道路上有没有东西挡着。

Visibility is settled; what remains is colour. The simplest local lighting model splits a surface's colour into three terms: ambient (a constant standing in for light bouncing everywhere), diffuse (Lambert: proportional to \(\mathbf n\cdot\mathbf l\), the dot product of 1.7, brighter the smaller the angle between normal and light) and specular (Blinn–Phong: \((\mathbf n\cdot\mathbf h)^m\), with \(\mathbf h\) the half-way vector between light and view directions; a larger \(m\) gives a smaller highlight). The normal \(\mathbf n\) comes from the mesh: one normal per face gives flat shading (the facets show); one normal per vertex, interpolated, makes the mesh look smooth. Computing the lighting at the vertices and interpolating the colour is Gouraud shading; interpolating the normal and computing the lighting per pixel is Phong shading, and only the latter can put a highlight between two vertices.

FIG 5 compares the two ways of rendering a sphere mesh over a checkered floor. The left image is the rasterization pipeline above; the middle one is ray tracing: one ray per pixel is intersected with the analytic sphere and the plane, and a shadow ray towards the light tells whether the sphere blocks that point; the right image is their difference (amplified 4 times). Raise the sphere resolution \(N\) and the mesh approaches the true sphere, so the two images converge and only the polygonal outline remains. Switch shadows on and the shadow on the floor lights up in the difference image, because the rasterizer knows only "this fragment's normal and light direction", not whether anything stands in the way.

\(\tilde{\mathbf p}=(x,y,z,1)\) 是齐次坐标（10.2）。模型矩阵 \(M={}^{w}T_{m}\in\mathrm{SE}(3)\)（5.3）把模型坐标变到世界；机器人的第 \(i\) 个连杆有 \(M_i={}^{w}T_{i}(\boldsymbol\theta)\)，就是 FK 的前 \(i\) 个因子之积（7.3）。视图矩阵 \(V=\operatorname{diag}(1,-1,-1,1)\,{}^{c}T_{w}\)，其中 \({}^{c}T_{w}=({}^{w}T_{c})^{-1}\) 是 OpenCV 外参（相机位姿的逆，10.3），对角阵把 OpenCV 的 \((x,y,z)\) 轴（右、下、前）换成 OpenGL 眼空间的轴（右、上、后）。视口里的 \(v\) 带负号，是因为 NDC 的 \(y\) 向上而像素 \(v\) 向下；这就是图形学管线里那一次"翻转"。

\(\tilde{\mathbf p}=(x,y,z,1)\) is a homogeneous coordinate (10.2). The model matrix \(M={}^{w}T_{m}\in\mathrm{SE}(3)\) (5.3) takes model coordinates to the world; link \(i\) of a robot has \(M_i={}^{w}T_{i}(\boldsymbol\theta)\), the product of the first \(i\) factors of FK (7.3). The view matrix is \(V=\operatorname{diag}(1,-1,-1,1)\,{}^{c}T_{w}\), where \({}^{c}T_{w}=({}^{w}T_{c})^{-1}\) is the OpenCV extrinsics (the inverse of the camera pose, 10.3) and the diagonal matrix turns OpenCV's \((x,y,z)\) axes (right, down, forward) into OpenGL's eye axes (right, up, backward). The minus sign on \(v\) in the viewport comes from NDC's \(y\) pointing up while the pixel's \(v\) points down; that is the one "flip" in the graphics pipeline.

设眼空间点 \((x_e,y_e,z_e)\)，深度 \(d=-z_e>0\)。OpenCV 针孔模型给出 \(u=f_x x_e/d+c_x\)；NDC 的 \(x_n=2u/w-1\)。要让除法恰好是除以 \(d\)，就令 \(w_c=d=-z_e\)（第四行 \((0,0,-1,0)\)），于是 \(x_c=x_nw_c=\tfrac{2f_x}{w}x_e+(1-\tfrac{2c_x}{w})z_e\)，\(y_c\) 同理（注意 \(y_n=1-2v/h\)，\(v=f_y y_e/z_e+c_y\)）。深度行取 \(z_c=Az_e+B\)，要求 \(d=n\) 时 \(z_n=-1\)、\(d=f\) 时 \(z_n=+1\)，解得 \(A=-\frac{f+n}{f-n}\)、\(B=-\frac{2fn}{f-n}\)，于是

Take an eye-space point \((x_e,y_e,z_e)\) with depth \(d=-z_e>0\). The OpenCV pinhole model gives \(u=f_x x_e/d+c_x\), and NDC has \(x_n=2u/w-1\). To make the division come out as a division by \(d\), set \(w_c=d=-z_e\) (the fourth row \((0,0,-1,0)\)); then \(x_c=x_nw_c=\tfrac{2f_x}{w}x_e+(1-\tfrac{2c_x}{w})z_e\), and \(y_c\) likewise (note \(y_n=1-2v/h\) and \(v=f_y y_e/z_e+c_y\)). Take the depth row as \(z_c=Az_e+B\) and require \(z_n=-1\) at \(d=n\) and \(z_n=+1\) at \(d=f\); this gives \(A=-\frac{f+n}{f-n}\) and \(B=-\frac{2fn}{f-n}\), so

右边是常见的"视场角 \(\varphi\)、宽高比 \(a=w/h\)"写法，它就是主点在图像中心、像素为正方形的特例。Direct3D、Metal、Vulkan 要求 \(z_n\in[0,1]\)，第三行换成 \((0,0,\tfrac{f}{n-f},\tfrac{nf}{n-f})\)；Vulkan 的 NDC \(y\) 向下，视口里就不用再翻 \(v\)。本课程每一张图都用 OpenGL 的约定，并在 FIG 4 里与 \(K\) 逐项对照，两条路线的像素差在 \(10^{-11}\) 以下。

The right-hand form is the common "field of view \(\varphi\), aspect ratio \(a=w/h\)" recipe: the special case with the principal point at the image centre and square pixels. Direct3D, Metal and Vulkan want \(z_n\in[0,1]\) and use the third row \((0,0,\tfrac{f}{n-f},\tfrac{nf}{n-f})\); Vulkan's NDC \(y\) points down, so its viewport does not flip \(v\). Every figure in this lesson uses the OpenGL convention and compares it entry by entry with \(K\) in FIG 4; the two routes agree to better than \(10^{-11}\) px.

\(z_n\) 对 \(1/d\) 是线性的，所以它对屏幕坐标也是仿射的（见下面的推导），这使得 z-buffer 可以像颜色一样用重心坐标插值深度，而不用逐像素做除法。代价是深度的分辨率：\(b\) 位定点深度缓冲在距离 \(d\) 处只能分辨 \(\Delta d\) 的间距，正比于 \(d^2/n\)。\(n=0.1\) m 时区间 \([n,2n]\) 就用掉一半的数值范围，\([n,1\text{ m}]\) 用掉 90%；把 \(n\) 缩小 10 倍，所有距离处的分辨率都差 10 倍（FIG 4 右图）。间距小于 \(\Delta d\) 的两个表面在缓冲里存成同一个值，谁赢取决于光栅化顺序和舍入，于是出现闪烁的 z-fighting。实践中近平面取尽量大、远平面不要无谓地大；现代引擎还常用反向 Z（把 \([n,f]\) 映射到 \(1\to 0\)）加浮点深度来均衡精度。

\(z_n\) is linear in \(1/d\), hence affine in the screen coordinates (derivation below), which is why the z-buffer can interpolate depth with barycentrics just like a colour, without a per-pixel division. The price is depth resolution: a \(b\)-bit fixed-point depth buffer can separate surfaces only by \(\Delta d\) at distance \(d\), proportional to \(d^2/n\). With \(n=0.1\) m the interval \([n,2n]\) already uses half of the numeric range and \([n,1\text{ m}]\) uses 90%; shrink \(n\) by 10 and the resolution is 10 times worse at every distance (the right plot of FIG 4). Two surfaces closer than \(\Delta d\) are stored as the same value, the winner then depends on rasterization order and rounding, and the result is flickering z-fighting. In practice take the near plane as large as the application allows and do not make the far plane needlessly large; modern engines also often use reversed Z (mapping \([n,f]\) to \(1\to0\)) with floating-point depth to even out the precision.

\(E_{12}(\mathbf v_0)\) 是整个三角形的有向面积的两倍，\(E_{12}(\mathbf p)\) 是 \(\mathbf p\) 与对边 \(\mathbf v_1\mathbf v_2\) 构成的子三角形的面积的两倍。在三角形内三者同号，\(\lambda_i\ge0\) 且 \(\sum\lambda_i=1\)；在顶点 \(\mathbf v_i\) 处 \(\lambda_i=1\)、其余为 0；对顶点值为仿射函数 \(f=\mathbf g\cdot\mathbf p+c\) 的量，\(\sum\lambda_if_i=f(\mathbf p)\) 精确成立（本课在 node 里验证过）。公式最右边说明，沿一行像素走一步，\(E\) 只需做一次加法，这是 Pineda（1988）并行光栅化的基础：硬件对包围盒里所有像素同时计算三个 \(E\)。朝向也藏在符号里：屏幕坐标（\(v\) 向下）里 \(E_{01}(\mathbf v_2)<0\) 对应逆时针即正面，\(>0\) 是背面，背面剔除就用这个符号。

像素中心恰好落在两个三角形的共享边上时，必须保证恰好一个三角形拥有它：都取（\(E\ge0\)）会画两次（半透明会变深），都不取（\(E>0\)）会留下一条缝。左上规则规定：边是"左边"（三角形在它的右侧）或"顶边"（三角形在它的下方）时，才算拥有边上的像素。我们在 node 里对 297 个随机的整数坐标四边形（沿对角线拆成两个三角形）验证了：左上规则下 0 个像素被画两次、0 个像素漏掉；"取等号"画了 193 个重复像素，"不取等号"留下 193 个洞。

\(E_{12}(\mathbf v_0)\) is twice the signed area of the whole triangle and \(E_{12}(\mathbf p)\) twice the area of the sub-triangle spanned by \(\mathbf p\) and the opposite edge \(\mathbf v_1\mathbf v_2\). Inside the triangle all three have the same sign, \(\lambda_i\ge0\) and \(\sum\lambda_i=1\); at vertex \(\mathbf v_i\) we get \(\lambda_i=1\) and the others 0; and for a quantity whose vertex values come from an affine function \(f=\mathbf g\cdot\mathbf p+c\), \(\sum\lambda_if_i=f(\mathbf p)\) holds exactly (checked numerically in node for this lesson). The last formula says that stepping one pixel along a row changes \(E\) by a single addition, the basis of Pineda's (1988) parallel rasterization: hardware evaluates the three \(E\)'s for all pixels of the bounding box at once. The winding is in the sign too: in screen coordinates (\(v\) down) \(E_{01}(\mathbf v_2)<0\) means counter-clockwise, i.e. front-facing, and \(>0\) back-facing; back-face culling uses exactly this sign.

When a pixel centre lies exactly on an edge shared by two triangles, exactly one of them must own it: taking it in both (\(E\ge0\)) draws it twice (a translucent surface gets darker), taking it in neither (\(E>0\)) leaves a crack. The top-left rule says an edge owns its pixels only if it is a "left edge" (the triangle lies to its right) or a "top edge" (the triangle lies below it). We checked this in node on 297 random integer-coordinate quadrilaterals split along a diagonal: with the top-left rule 0 pixels are drawn twice and 0 are missed; "\(\ge\)" draws 193 pixels twice and "\(>\)" leaves 193 holes.

三角形所在的平面上，眼空间点满足 \(\mathbf n\cdot\mathbf p_e=c\)。写 \(\mathbf p_e=w\,(\tilde x,\tilde y,-1)\)，其中 \(\tilde x,\tilde y\) 是屏幕坐标的仿射函数（\(w=d\) 是深度），代入得 \(w\,\mathbf n\cdot(\tilde x,\tilde y,-1)=c\)，即 \(1/w=\mathbf n\cdot(\tilde x,\tilde y,-1)/c\) 是屏幕坐标的仿射函数。若属性 \(f\) 在表面上是眼空间坐标的仿射函数 \(f=f_0+\mathbf g\cdot\mathbf p_e\)，则 \(f/w=f_0/w+\mathbf g\cdot(\tilde x,\tilde y,-1)\) 也是屏幕坐标的仿射函数。仿射函数可以用重心坐标精确插值，于是

On the triangle's plane the eye-space points satisfy \(\mathbf n\cdot\mathbf p_e=c\). Write \(\mathbf p_e=w\,(\tilde x,\tilde y,-1)\), where \(\tilde x,\tilde y\) are affine functions of the screen coordinates and \(w=d\) is the depth; substituting gives \(w\,\mathbf n\cdot(\tilde x,\tilde y,-1)=c\), i.e. \(1/w=\mathbf n\cdot(\tilde x,\tilde y,-1)/c\) is an affine function of the screen coordinates. If an attribute \(f\) is affine in eye-space coordinates on the surface, \(f=f_0+\mathbf g\cdot\mathbf p_e\), then \(f/w=f_0/w+\mathbf g\cdot(\tilde x,\tilde y,-1)\) is also affine in the screen coordinates. Affine functions are interpolated exactly by barycentric coordinates, so

FIG 3 在 node 里与光线投射真值逐像素比较：透视校正的最大误差为 \(1\times10^{-14}\) 个格子，线性插值是 8.4 个格子。对近平面裁剪后新生成的顶点同样成立：裁剪在齐次裁剪空间里线性插值属性，我们也验证了裁剪后地面的世界坐标与光线投射一致（误差 \(3\times10^{-13}\)）。

FIG 3 was compared pixel by pixel against the ray-cast truth in node: the perspective-correct maximum error is \(1\times10^{-14}\) squares while the affine version is off by 8.4 squares. The same holds for the vertices created by near-plane clipping: clipping interpolates attributes linearly in homogeneous clip space, and we verified that the world coordinates of a clipped floor agree with ray casting (error \(3\times10^{-13}\)).

法向是协变量：表面上的切向 \(\mathbf t\) 按 \(\mathbf t'=M\mathbf t\) 变换，而要保持 \(\mathbf n'\cdot\mathbf t'=0\)，需要 \(\mathbf n'=M^{-\top}\mathbf n\)。当 \(M\) 是刚体运动（如机器人连杆）时 \(M^{-\top}\!=M\) 的旋转部分，直接乘 \(M\) 就行；一旦有非均匀缩放，就必须用逆转置（例：\(S=\operatorname{diag}(2,1,1)\) 作用在法向 \((1,1,0)\) 上应得 \((\tfrac12,1,0)\propto(1,2,0)\)，直接乘 \(S\) 会得到 \((2,1,0)\)）。光线追踪与光栅化的复杂度对比：光栅化约 \(O(T+\text{覆盖的像素})\)，光线追踪暴力法 \(O(PT)\)，用 BVH 约 \(O(P\log T)\)，其中 \(P\) 是像素数、\(T\) 是三角形数。统一两者的是渲染方程（Kajiya，1986）

Normals are covariant quantities: a tangent \(\mathbf t\) on the surface transforms as \(\mathbf t'=M\mathbf t\), and keeping \(\mathbf n'\cdot\mathbf t'=0\) requires \(\mathbf n'=M^{-\top}\mathbf n\). For a rigid motion \(M\) (like a robot link) \(M^{-\top}\) equals the rotation part of \(M\) and plain multiplication is fine; as soon as there is non-uniform scaling the inverse transpose is mandatory (example: \(S=\operatorname{diag}(2,1,1)\) acting on the normal \((1,1,0)\) must give \((\tfrac12,1,0)\propto(1,2,0)\), while multiplying by \(S\) gives \((2,1,0)\)). Complexity: rasterization costs about \(O(T+\text{covered pixels})\); brute-force ray tracing \(O(PT)\) and with a BVH about \(O(P\log T)\), where \(P\) is the number of pixels and \(T\) the number of triangles. What unifies the two is the rendering equation (Kajiya, 1986)

出射光 = 自发光 + 半球上"入射光 × 材质 × 余弦"的积分。Lambert 是 \(f_r=k_d/\pi\)（我们把 \(\pi\) 吸收进 \(k_d\)）；上面的 Phong 模型只对点光源或方向光求这个积分，不含间接光。路径追踪用蒙特卡洛估计整个积分，光栅化管线则靠阴影贴图（Williams，1978：先从光源处渲染一张深度图，着色时比较深度）、环境贴图等技巧近似其中的局部项。可微渲染（SoftRas、nvdiffrast、PyTorch3D）把"覆盖测试"做成可微的，使渲染结果对顶点位置、材质、相机位姿有梯度。12.4 会看到两条后续路线：NeRF 用沿光线的体渲染积分代替"找最近表面"，3D Gaussian Splatting 则把大量带颜色的三维高斯投影成屏幕上的椭圆再按深度排序混合，也就是对高斯做光栅化。

Outgoing light = emitted light + the hemisphere integral of "incoming light × material × cosine". Lambert is \(f_r=k_d/\pi\) (we absorb the \(\pi\) into \(k_d\)); the Phong model above evaluates this integral only for point or directional lights, with no indirect light. Path tracing estimates the whole integral by Monte Carlo, while the rasterization pipeline approximates parts of it with tricks such as shadow maps (Williams, 1978: render a depth map from the light, then compare depths while shading) and environment maps. Differentiable rendering (SoftRas, nvdiffrast, PyTorch3D) makes the coverage test differentiable so that the rendered image has gradients with respect to vertex positions, materials and the camera pose. 12.4 continues along two routes: NeRF replaces "find the nearest surface" with a volume-rendering integral along each ray, and 3D Gaussian Splatting projects many coloured 3D Gaussians to ellipses on the screen and blends them in depth order, i.e. it rasterizes Gaussians.

最直接的应用是用虚拟相机拍训练数据。FIG 6 是一个桌面抓取场景的缩影：一块桌面和四个物体，一台斜向下看的虚拟相机。同一次渲染输出三种东西：RGB；深度图，由 z-buffer 的 \(d\) 经 \(z=\frac{fn}{f-d(f-n)}\) 还原成米（在 node 里和光线投射的深度对照，相对误差 \(10^{-6}\)）；实例分割，即每个像素属于哪个物体。从分割图取每个物体的最小外接矩形，目标检测的边框标注也是免费的（RGB 图里的彩色矩形）。

仿真到真实 (sim-to-real) 的难点是域差距：渲染的图像和真实照片在光照、材质、传感器噪声上不一样，训练在仿真上的网络到真机上会失灵。域随机化（Tobin 等，2017）的办法是在仿真里把这些因素随机变化，让网络见到足够宽的分布，使真实世界成为其中一个样本。拖动"随机化强度"，下方 4 张缩略图是同一个场景的四次随机采样：光照方向和强度、物体颜色、相机位姿和视场角、传感器噪声都在变，而标注（边框、分割、深度）始终与图像严格对应。深度图里的噪声模型（方差随 \(z^2\) 增大，并有少量缺失像素）只是示意，真实传感器的噪声形态因类型而异。

The most direct use is shooting training data with a virtual camera. FIG 6 is a miniature of a tabletop grasping scene: a table, four objects and a virtual camera looking obliquely down. One render gives three outputs: RGB; a depth image, where the z-buffer's \(d\) is converted back to metres by \(z=\frac{fn}{f-d(f-n)}\) (checked in node against ray-cast depth, relative error \(10^{-6}\)); and an instance segmentation, i.e. which object each pixel belongs to. Taking each object's bounding rectangle from the segmentation gives object-detection box labels for free (the coloured rectangles in the RGB image).

The difficulty of sim-to-real transfer is the domain gap: rendered images differ from real photographs in lighting, materials and sensor noise, and a network trained only in simulation can fail on the real robot. Domain randomization (Tobin et al., 2017) randomizes these factors in simulation so that the network sees such a broad distribution that the real world looks like just one more sample. Drag the "randomization strength" and the four thumbnails below show four random draws of the same scene: lighting direction and strength, object colours, camera pose and field of view and sensor noise all vary, while the labels (boxes, segmentation, depth) stay exactly aligned with the image. The depth-noise model (variance growing with \(z^2\), a few dropped pixels) is only illustrative; the noise of real sensors depends on the sensor type.