VoxPoser turns language into 3D spatial objectives for a motion planner, rather than direct actuator commands. Inspect the map-to-planner interface before blaming a prompt: in our AI-assisted CPU check of its released neighborhood helper, a grid side length of 49 produced zero candidate voxels and an empty-array selection error; size 100 produced 124 candidates at the grid center. This isolated synthetic check did not run the complete planner, language model or robot.
If a demonstration looks successful but a reproduction moves toward the wrong place, inspect the value maps and their coordinate frame before treating the failure as a language-model problem.
From object language to a spatial objective
The VoxPoser project combines language-model-generated programs with visual grounding to construct three-dimensional value maps. These can express where motion is desirable or undesirable, along with preferences involving rotation, velocity and gripper behavior. A planner uses the composed objectives to produce motion waypoints. Visual feedback lets the system update the spatial problem as the scene changes.
The paper presents this as a way to compose robot manipulation behavior without collecting task-specific training data for every instruction. That research claim concerns the proposed system and its demonstrations. It does not mean an unfamiliar camera, robot or object automatically satisfies the perception and control assumptions.
For a hypothetical instruction to move a cup around a box, one map might favor the destination while another discourages passing through the box. The result is a planning objective over space, not a textual list saying “go left, then forward.” The language program defines how the objective is constructed; the planner must still find useful motion under the available model and controller.
Inspect the maps before tuning the prompt
Several different errors can produce an apparently similar bad path. A wrong object mask can put an avoidance region in the wrong place. An incorrect camera transform can shift an otherwise sensible map. A valid objective can still be poorly served by the planner or controller. The following diagnostic table is an editorial procedure, not a reported VoxPoser evaluation.
| Boundary | Evidence to inspect | Failure that the check can distinguish |
|---|---|---|
| Language to grounded object | Referenced object and observed mask or point cloud | Wrong referent versus an inappropriate spatial objective |
| Grounded object to map | Map location and axes in the workspace frame | Coordinate error versus a correctly placed preference |
| Maps to waypoints | Attractive regions, avoidance regions and proposed path together | Objective conflict versus planner behavior |
| Waypoints to execution | Command convention, robot state and observed motion | Valid waypoint plan versus controller or tracking error |
Try a small, deliberately unambiguous scene first. Move one obstacle while keeping the instruction unchanged, and inspect whether the relevant map moves with it. Keep the old and new observations rather than retaining only the final path. If the map stays fixed, a faster planner will not solve the stale-perception problem.
Then change only the requested destination. This separates sensitivity to the instruction from sensitivity to the scene. Those scene-change tests remain suggestions for a future reproduction. The separate CPU check below uses a constructed cost volume; it does not generate an observed-scene map, a complete trajectory or a robot measurement.
A smaller voxel grid can leave no next-step candidates
Before changing a language prompt, check whether the planner has any neighbors to score. In the released planner at revision e3a4c9e, the neighborhood radius is int(2 * map_size / 100). The helper builds offsets around the current voxel, removes the zero offset, clips positions to the grid and removes duplicates. Its object-centric branch permits only xy offsets because that path assumes planar pushing.
We directly executed this unchanged helper and the three score-and-select statements in an AI-assisted CPU check with Python 3.12.14 and NumPy 2.3.5. We extracted the inspected syntax nodes instead of importing the full module. Inputs were integer grid centers and synthetic squared-distance costs, with an artificial minimum three voxels along positive x. No observed scene, language model, complete planner, SciPy preprocessing or robot was run.
| Grid side length | Integer radius | 3D candidates at center | Planar candidates at center | Isolated selection result |
|---|---|---|---|---|
| 49 | 0 | 0 | 0 | ValueError from np.argmin on an empty score array |
| 50 | 1 | 26 | 8 | Selects (26, 25, 25) in either mode |
| 100 | 2 | 124 | 24 | Selects (52, 50, 50) in either mode |
At size 49, the only initial offset is zero. Removing it leaves an array of shape (0, 3), so there is no lowest-cost neighbor to choose. The failure happens before the quality of the synthetic cost preference can matter. At 50 and 100, candidate generation and the isolated selection succeeded. Size 100 is also present in the checked-in RLBench configuration. This is not evidence that 50 is a supported minimum for the entire system, or a recommendation to change an existing deployment to that resolution.
Grid edges behave differently from interior points. At (0, 0, 0) in a size-100 grid, clipping left 27 unique 3D candidates, including the current voxel. Although the helper removes the zero offset before clipping, other offsets can clip back onto the origin. Thus candidate count and whether staying in place is possible both deserve inspection at workspace boundaries.
The compact NumPy reproduction below generated exactly the same candidate arrays as the official helper in six center cases and four corner controls. It computes scores directly at candidates rather than constructing the full cost volume. Run it with NumPy available; the size-49 error is caught and printed intentionally.
import numpy as np
def candidates(n, planar=False, corner=False):
# Compact equivalent of the inspected neighborhood helper.
half = int(2 * n / 100)
offsets = np.arange(-half, half + 1)
z = [0] if planar else offsets
grid = np.array(np.meshgrid(offsets, offsets, z)).T.reshape(-1, 3)
grid = grid[np.any(grid != [0, 0, 0], axis=1)]
current = np.array([0, 0, 0] if corner else [n // 2] * 3)
points = np.unique(np.clip(current + grid, 0, n - 1), axis=0)
goal = current + [3, 0, 0] # artificial cost minimum, no observed scene
return points, goal
for n in (49, 50, 100):
for planar in (False, True):
points, goal = candidates(n, planar)
scores = np.sum((points - goal) ** 2, axis=1)
try:
selected = points[np.argmin(scores)].tolist()
except ValueError as err:
selected = type(err).__name__
print(n, "planar" if planar else "3D", len(points), selected)
points, _ = candidates(100, corner=True)
print("100 corner:", len(points), "includes origin:", bool(np.any(np.all(points == 0, axis=1))))
Record map_size, the integer radius, candidate count and object-centric mode alongside a failing trace. Restoring size 100 recovered a nonempty candidate set and one isolated selection in this check; it did not demonstrate a completed path. Distance transforms, map normalization, smoothing, stopping criteria, postprocessing and collision validation remain outside the execution. Reducing memory by shrinking a grid can change planner behavior, even when the instruction and desired location are unchanged.
The released simulator supplies part of the perception contract
The official repository exposes separate language-program, environment-interface, planner, controller and dynamics-model components. Its released RLBench implementation uses simulator observations and masks. The README explicitly says that the release does not contain the real-world perception pipeline. Treat that as a material reproduction boundary, not a minor installation detail.
A simulator can identify the relevant object without reproducing the same uncertainty as a physical camera pipeline. Replacing that input requires deciding how detections, masks, depth and coordinate transforms reach the map-building interface. A correct simulator example therefore does not validate the replacement perception stack.
Before connecting another environment, document which component owns each transform. For example, the camera-to-workspace transform should not be applied once while constructing points and then applied again while sending waypoints. The same principle applies to rotation conventions and gripper state. A visually plausible map can hide an interface mismatch until execution.
Pin the code and environment versions used for any reproduction. The repository is an implementation reference; a successful dependency installation alone is not evidence that its planner, perception assumptions and robot controller agree.
Choose the representation for the relation you need
Dense workspace maps make it natural to inspect regions of attraction and avoidance. Another formulation may represent a task through a smaller set of geometric relations. The ReKep keypoint-constraint guide explains that adjacent approach. This is a representation choice, not a benchmark claim that one system is universally more accurate.
Ask what would make a failure legible to the person debugging it: a misplaced spatial region, an unsatisfied relation between keypoints, or an incorrectly executed waypoint. Choose an experiment that exposes that specific boundary. VoxPoser’s useful contribution is a visible route from language to planner objectives; reliable execution still requires the grounding, geometry and control interfaces to agree.