SAPIEN’s rendered Position texture contains a camera-space 3D position in its first three channels and a z-buffer value in its fourth. The position uses graphics-camera axes: x points right, y points up and negative z points forward. That differs from the camera pose convention used elsewhere in SAPIEN, where x is forward, y is left and z is up.
To place these points in the world, use the camera’s model matrix after synchronizing the scene with the renderer. The matrix maps graphics-camera coordinates to SAPIEN world coordinates, so the texture’s xyz values can be transformed directly through that mapping.
The official camera tutorial, checked on October 16, 2026, supplies this data contract and the transformation. The explanation below follows its text and code; no scene was rendered or point cloud measured for this article.
One camera appears in two coordinate conventions
When the tutorial constructs a camera pose, its rotation columns describe forward, left and up. Those are the local x, y and z directions of the SAPIEN camera entity. A mounted camera also has a local pose relative to its parent entity, so moving the parent changes the camera’s world pose.
The renderer’s Position texture uses a different basis for the same view. Its x direction points right, its y direction points upward, and its z direction points backward. A visible point in front of the camera therefore has a negative z coordinate in this texture.
| Representation | x direction | y direction | Forward direction |
|---|---|---|---|
| SAPIEN camera pose | Forward | Left | Positive x |
| Graphics-camera Position xyz | Right | Up | Negative z |
A point directly ahead might consequently be represented as (0, 0, -2) in the Position texture. Treating those three values as coordinates in the camera entity’s forward-left-up basis would send the point along the wrong axis. The model matrix exists to connect the graphics representation to the world representation.
Separate xyz, depth and the fourth channel
The texture has shape [H, W, 4]. Each pixel carries (x, y, z, render_depth), but the last name refers to a rendering z-buffer value. The tutorial identifies a fourth-channel value of 1 with a position beyond the camera’s far plane and uses position[..., 3] < 1 to select points for the cloud.
Depth comes from a different operation: -position[..., 2]. Since visible points are forward along negative z, negating that coordinate yields positive depth along the viewing direction. The fourth channel is used by the tutorial’s validity mask; it is not the value negated to obtain depth.
This distinction also separates axial depth from the length of the 3D camera-to-point vector. For a point off the central viewing axis, its x and y coordinates contribute to that vector’s length, while the tutorial’s depth expression still uses only negative z.
Apply the same validity mask when selecting colors. The point and color arrays describe corresponding pixels before filtering, so preserving that correspondence gives each retained point its own pixel’s color.
Synchronize the renderer before reading its transform
The tutorial’s rendering sequence advances physics, synchronizes poses with scene.update_render(), and submits the camera render with camera.take_picture(). It then reads the rendered buffers. It specifically requires camera.get_model_matrix() to be called after scene.update_render().
That order connects the point positions with the pose used by the rendering system. A camera attached to a moving entity needs the transform for that rendered view, rather than a matrix retained from an earlier pose.
For a model matrix whose upper-left block is R and whose translation column is t, the point transformation is:
column-vector convention: p_world = R p_camera + t
row-vector point array: P_world = P_camera R^T + t
The tutorial uses the second form because its selected points are rows of a NumPy array. Transposing R makes row-vector multiplication express the same rotation as the usual column-vector formula. The translation is then added to every point.
The matrix already maps from the graphics-camera basis. An extra manual graphics-to-robotics axis swap before this multiplication would apply another basis change to data that the matrix expects in its original graphics convention.
A worked point shows the axis mapping
Consider an illustrative camera pose with world position t = (1, 2, 3). Let the camera face along world positive x, with its left direction along world positive y and its up direction along world positive z.
For this pose, graphics-camera right maps to world negative y, graphics-camera up maps to world positive z, and graphics-camera backward maps to world negative x. The corresponding graphics-to-world rotation is:
R = [ 0 0 -1 ]
[-1 0 0 ]
[ 0 1 0 ]
p_camera = (0, 0, -2)
R p_camera = (2, 0, 0)
p_world = (2, 0, 0) + (1, 2, 3) = (3, 2, 3)
The point’s camera depth is 2, and its world position is two units forward from the camera. For an off-axis example (0.5, 0.2, -2), the same rotation and translation produce (3, 1.5, 3.2): camera-right moves toward world negative y, while camera-up increases world z.
Both calculations assume the sample passed the fourth-channel validity test. A sample with a z-buffer value of 1 is excluded by the tutorial’s mask before its xyz values enter this geometry calculation.
Keep mounting and point conversion in their respective frames
The tutorial distinguishes changing the camera’s local pose from moving its parent entity. camera.set_local_pose changes the pose relative to that parent; camera.entity.set_pose changes the parent’s pose. For a mounted camera, both relationships contribute to where the view sits in the world.
Once those poses have been synchronized to the renderer, get_model_matrix() supplies the graphics-camera-to-world mapping used for the captured Position data. The practical sequence is therefore well-defined: establish the camera and mount poses, synchronize, render, mask the buffers and transform the retained xyz points. Each stage preserves a specific relationship between image pixels, camera axes and world geometry.
If these depth values later become a training dataset, their stored representation adds another decision. The LeRobot depth-video guide explains units and quantizer metadata. Defining which geometric value a pixel represents comes first; choosing how to encode that value is the next step.