RoboPoint predicts image points that fit a language instruction. Turning a point into a robot target still requires aligned depth, camera geometry and the robot-frame transform. In our AI-assisted synthetic projection check, treating cropped and scaled coordinates as raw pixels shifted the calculated target by 0.732393 meters; using a consistent mapping removed that difference. No RoboPoint model, camera or robot was run. The calculation illustrates the interface error, not expected real-world accuracy.
The RoboPoint project learns this spatial prediction task using procedurally generated 3D scenes and instruction–point examples. Its deployment pipeline projects predicted 2D points into 3D using depth and passes targets to a motion planner. That division is the key to evaluating it: judge the point prediction separately from projection and execution.
Keep three kinds of error separate
Suppose the instruction is “place the cup in the free space to the right of the plate.” The image point may correctly identify the requested region, yet its depth sample may fall on the plate’s edge. Alternatively, the depth may be valid while the camera-to-robot transform is outdated. Both cases can produce a wrong robot target from a reasonable image prediction.
Treat the pipeline as three checks. First, overlay the predicted point on the exact image submitted to the model. Second, inspect the corresponding depth and reconstructed camera-space position. Third, inspect the robot-frame target and the planned motion. This hypothetical placement example shows why final task success alone is a poor diagnostic: it compresses failures from several stages into one number.
| Stage | Question to answer | Failure that can remain hidden |
|---|---|---|
| Language and image point | Is the selected region consistent with the instruction? | The point lands on the correct object but the wrong side |
| Pixel and depth alignment | Does depth describe the same scene location? | Resizing, cropping or camera mismatch selects a different surface |
| Camera-to-robot transform | Is the reconstructed point expressed in the execution frame? | A stale camera pose shifts every target |
| Motion planning | Can the robot reach the target with a suitable approach? | A reachable point still requires a colliding or unsuitable gripper pose |
Undo image preprocessing before using depth
Models and cameras need not use the same image dimensions. If an image was cropped, resized or padded for inference, retain that mapping. Convert the predicted output into coordinates on the calibrated depth-aligned image before sampling depth. Do not assume that a coordinate pair is already expressed in raw camera pixels; follow the output convention of the particular implementation you use.
For a calibrated pinhole camera, the geometry is straightforward once that correspondence is correct. With pixel coordinates u, v, focal lengths fx, fy, principal point cx, cy, and depth z measured along the camera’s optical axis, the camera-frame coordinates are x = (u − cx) × z / fx, y = (v − cy) × z / fy, and z. If the sensor supplies distance along a ray instead, that value must first be converted to the required depth convention. Lens distortion must also be handled consistently with the calibration.
These equations explain the projection stage; they do not specify a RoboPoint output format. The constructed calculation below exercises this geometry without running the model or a sensor. They show why an apparently small bookkeeping mistake matters. Applying the original camera intrinsics to resized pixel coordinates changes the ray. Sampling a missing or reflective depth region supplies no trustworthy distance at all.
Reproduce a crop-and-scale error before connecting the planner
We ran an AI-assisted NumPy calculation of the projection boundary with Python 3.12.14 and NumPy 2.3.5. It used constructed coordinates and optical-axis depth; no RoboPoint inference, camera image, depth sensor, OpenCV function or robot motion was executed. The OpenCV 4.13 camera-geometry reference defines the pinhole intrinsics and how their pixel scale changes with image scale.
Our calibrated-image dimensions were 640 × 480, with fx = fy = 600, cx = 320, cy = 240, and depth 1.2 meters. We declared the coordinate mapping submitted = 0.5 * (raw − [160, 120]): a crop origin of (160, 120) and half-scale coordinates. A synthetic point (140, 90) in the submitted coordinate system therefore corresponds to raw pixel (440, 300).
This is an explicitly selected affine mapping. No image was resampled, and no library’s pixel-center interpolation rule was tested. It is also not a claim about RoboPoint’s output normalization. A real adapter must retain its own exact preprocessing transform.
| How the point is interpreted | Camera-frame target in meters | Separation from the consistent target |
|---|---|---|
| Undo scale and crop before applying original intrinsics | (0.24, 0.12, 1.20) |
0 |
Treat submitted (140, 90) as raw pixels |
(−0.36, −0.30, 1.20) |
0.732393 m |
| Undo scale but omit the crop offset | (−0.08, −0.12, 1.20) |
0.400000 m |
| Keep submitted coordinates and transform intrinsics consistently | (0.24, 0.12, 1.20) |
0 |
The last route uses fx = fy = 300, cx = 80, and cy = 60. It agrees with undoing the coordinate transform first. Both approaches are valid for this defined mapping; mixing a point from one coordinate system with intrinsics from the other is the error. These separations are calculated target differences, not measured robot error or estimates of typical deployment accuracy.
Run this NumPy snippet to reproduce the four rows. It also converts an explicitly known synthetic depth of 1,200 millimeters to meters and rejects nonpositive or nonfinite depth. The conversion relies on a declared unit; the code does not infer a sensor’s unit from the magnitude.
import numpy as np
def point(pixel, z, intrinsics):
fx, fy, cx, cy = intrinsics
if not np.isfinite(z) or z <= 0:
raise ValueError("finite positive optical-axis depth required")
u, v = pixel
return np.array([(u-cx)*z/fx, (v-cy)*z/fy, z])
# Declared affine coordinate mapping; no actual image is resized.
k = np.array([600., 600., 320., 240.])
offset = np.array([160., 120.])
scale = .5
submitted = np.array([140., 90.])
raw = submitted / scale + offset
correct = point(raw, 1.2, k)
direct = point(submitted, 1.2, k)
no_crop = point(submitted / scale, 1.2, k)
adjusted_k = np.r_[k[:2]*scale, (k[2:]-offset)*scale]
via_adjusted_k = point(submitted, 1.2, adjusted_k)
print("raw pixel:", raw)
for name, value in [("correct",correct),("submitted as raw",direct),
("crop offset omitted",no_crop),("adjusted K",via_adjusted_k)]:
print(name, value, "separation m:", round(float(np.linalg.norm(value-correct)),6))
assert np.allclose(correct, via_adjusted_k)
print("known 1200 mm converted:", point(raw, 1200/1000, k))
for z in [0., -1., np.nan]:
try:
point(raw, z, k)
except ValueError:
print("rejected depth:", z)
Across 100 seeded points within the constructed crop, the two consistent routes agreed within 1e-12 meters, and reprojection recovered the original pixels. Feeding 1,200 directly into the meter-valued calculation instead multiplied all three coordinates by 1,000. Zero, negative, NaN and infinite depths were rejected by our local guard.
That guard is deliberately narrow: it does not check confidence, depth/RGB alignment, image bounds, calibration freshness, distortion, workspace limits or reachability. The experiment isolates coordinate bookkeeping under an undistorted pinhole assumption. Save the preprocessing transform and depth convention alongside a prediction, then visualize the reconstructed point before sending it to any execution primitive.
A target position leaves the approach unresolved
Once the camera-frame point is available, transform it using the calibrated camera-to-robot relationship. For a moving wrist camera, use the camera pose associated with the observation, not a convenient pose from a later time. Then decide what the point means: a contact location, a placement position or a reference for a separate motion primitive.
A placement routine may need an offset for the held object’s shape and a chosen gripper orientation. A contact routine may need an approach direction and a force limit. Those decisions are not supplied merely by a correct 2D location. Keep them in a named execution primitive whose assumptions can be tested independently.
Before enabling a physical action, visualize the robot-frame point against the known scene and reject invalid depth, out-of-workspace targets and impossible plans. An overlay that looks right on the camera image is necessary evidence for grounding, but it is not evidence that the complete motion is feasible.
Measure the interface you actually want to improve
For a local evaluation, preserve the instruction, submitted image, preprocessing mapping, predicted point, depth validity, calibrated 3D target and planner outcome. Label whether a failure began with language grounding, projection, calibration or execution. The robot action-data guide provides broader episode-recording context; these intermediate values make a point-based system diagnosable.
The RoboPoint paper evaluates its spatial prediction approach and downstream robot use. Its reported gains belong to those evaluation conditions; they do not certify your camera alignment or planner. Choose this kind of model when instruction-conditioned spatial targets are the missing component, and retain the geometry and control stages as explicit parts of the system. The NumPy calculation above reconstructs targets from synthetic pixels and depth. No RoboPoint model, sensor-based reconstruction or robot motion was run for this article.