A low relational-keypoint cost means a planned pose scores well under the encoded objective. It does not establish that every task requirement is satisfied. It does not, by itself, mean the robot grasped the object, avoided every collision or completed the task. When a ReKep-style plan looks right but execution fails, inspect the boundary in order: the chosen points, the stage and path constraints, the collision representation, then the conversion from end-effector pose to robot motion.
ReKep represents a desired relation among tracked 3D points as a numerical cost. A sequence of such relations can guide a multi-stage motion. The authors’ project page describes constraints generated from an RGB-D observation and a language instruction, then optimized into end-effector poses. The attraction is specific: “keep these parts in this spatial relationship” can be expressed without hand-writing a complete joint trajectory for every new scene. The result is still a task-space target, not an executed contact or a low-level control guarantee.
A plug can line up on screen and still miss the socket
Consider a hypothetical plug-insertion task. A planner sees a point at the plug tip and another at the socket opening. It selects a pose that brings those points together. In a visualization, the endpoints may coincide. Yet the plug can approach at the wrong angle, sweep its cable into an obstacle, or reach the opening while the gripper no longer holds it. The point-pair relation was satisfied; the physical job was not.
This example suggests three separate questions. Does the subgoal define enough geometry to identify a useful end pose, including orientation? Does the path preserve the clearance and approach direction needed before that pose? Does the transition to insertion occur only after the grasp and alignment are actually established? A solver cannot infer an omitted requirement merely because its objective reaches a low value. Adding more points may help describe orientation, but each point must also remain visible, correctly associated with the object, and expressed in a consistent frame.
That is where relational constraints are useful. They make the intended relation inspectable. If the plan tilts the plug, you can ask which keypoints or cost terms permitted that tilt. The constraint is a compact specification to audit, not a certificate that the task will succeed.
Find the boundary where a good-looking plan breaks
The table is a proposed diagnosis for a local system, not a set of failures measured for this article. Preserve one failed rollout with camera frames, tracked 3D points, per-stage costs, proposed end-effector poses, measured robot state and task outcome on a shared timeline. Then locate the first disagreement.
| What appears to happen | First evidence to inspect | Likely next check |
|---|---|---|
| The proposed pose fits the constraint, but the object relation is wrong in the image | Point identity, depth, camera transform and tracking continuity | Correct the point association or frame before tuning the optimizer |
| The end pose is sensible, but the arm clips an obstacle on the way | Path constraints and the collision or signed-distance representation | Check whether the obstacle, robot and grasped object are represented and masked correctly |
| The planned pose is clear, but the wrist stops short or takes an awkward route | Inverse-kinematics feasibility, joint limits and the controller’s actual tracking | Test reachable poses and record commanded versus measured motion |
| Geometry reaches the intended relation, but the plug slips or never seats | Gripper state, contact event and stage-transition evidence | Separate grasp or insertion failure from the keypoint objective |
This ordering prevents a common repair mistake: changing language prompts or point costs when the tracked point jumped between frames, or changing a low-level controller when the planner never encoded clearance. A constraint score should be kept beside physical outcome. The EmbodiedSkills analysis covers the related problem of verifying a skill outcome before advancing a sequence. A locally defined success check might require both a completed insertion and a stable final state; its exact test depends on the hardware and task.
The released demo supplies information a robot must estimate
The official ReKep repository calls its released code an OmniGibson demo. Its README says that the real-world perception pipeline is not included: the demo obtains keypoint, mask and signed-distance information directly from simulation. That is a material boundary for anyone adapting it to a camera-and-robot setup. A demo command completing in simulation does not prove that the same observations can be recovered from live sensors.
The README also warns that task-space plans can be kinematically difficult to execute. Treat the perception and controller interfaces as things to validate on the intended robot.
For the hypothetical plug task, a bounded integration test could move the socket between observations while leaving the robot stationary. Do the tracked socket points follow the same physical feature? Does the collision map retain the fixture while excluding the robot’s own arm? Does the optimizer revise the approach path rather than only its final pose? Only after those checks would a low-speed robot trial tell you whether the pose interface and grasp are adequate. None of these tests was run for this article.
Stop at a reproducible failure label
For each failed attempt, save the stage name, constraint values, keypoint tracks, collision-field version, candidate pose, inverse-kinematics result, controller command, measured state and physical outcome. Label the earliest boundary that failed. If the wrong point was tracked, fix perception. If the point and target pose are correct but the route is blocked, fix the path and environment representation. If the pose is feasible but the object slips, investigate grasp and contact.
This is the practical value of a relational-keypoint formulation: the plan exposes what it believed had to be true. ReKep’s reported research system shows multi-stage, closed-loop manipulation on its authors’ platforms; the released simulation demo leaves substantial sensing and control work for a new robot. A constraint-satisfying trajectory is evidence about the encoded relation. Task completion needs separate evidence from the physical rollout. Sources checked September 29, 2026.